Hysteresis waveshaping

ABSTRACT

This invention provides a signal processing and signal synthesis technique from a family of signal processing and signal synthesis techniques designed to readily interwork or be used individually in creating new forms of rich musical timbres. Synthetic hysteresis waveshaping is used to create naturally appealing dynamic spectral variations from an input signal of varying amplitude and waveshape. Waveshaping curves, waveshaping parameters, hysteresis characteristics and hysteresis parameters may be recalled from stored program control or modulated in real-time by arbitrary control signals, including those derived from the original input signal. The invention may be used individually or in conjunction with other signal processing and signal synthesis techniques in creating new forms of rich musical timbres. The invention may also be used in spatially-distributed timbre construction.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.09/812,400, filed Mar. 19, 2001, which is a division of U.S. applicationSer. No. 09/313,533, filed May 15, 1999, now U.S. Pat. No. 6,610,917,issued Aug. 26, 2003, which claims benefit of priority of U.S.provisional application Ser. No. 60/085,713, filed May 15, 1998.

BACKGROUND OF THE INVENTION

1. Field of Invention

This invention relates to musical instrument performance systems andenvironments, and in particular to the combination of novel instrumententities built from synergistic arrangements of traditional and novelinstrument elements, and the interconnection of said instrument entitiesutilizing generalized interface entities to signal routing, processing,and synthesis entities built from synergistic combinations oftraditional and novel architectures, processes, and methodologies. Thesystems and methods herein are intended to make possible a newgeneration of musical instrument products with enhanced capabilities andsounds, new semiotic-oriented performance capabilities, and richcomposition and recording environments.

2. Background

There has been considerable advancement in music technology in the lastseveral decades, but recent innovations driven by mass-market forceshave narrowed the range of possibilities for commercially availableinstruments and the ways in which new recorded and performed music arebeing explored. Audio samples of diverse instruments, advanced signalprocessing power, improved fidelity, the MIDI control interface,sequencers, and music workstations are important assets but, togetherwith the ways synthesizers, signal processing systems, and instrumentcontrollers have come to be designed, the channel of innovation isfocused on a relatively narrow conceptual range that will consume asmuch rework and refinement energy as can be allotted. A few modernoutlier innovations have appeared, such as the Roland COSM signalprocessing methods, Yahama VL1 model-based synthesis methods, andBuchla's and Starr Switch alternative MIDI controllers, but due to thefocused drive of the mainstream these exceptions are largely orphaned intheir application.

What is needed is some reach into the souls (rather than make samples)of deep non-Western and Western instruments, a recasting of the nowinstitutionalized signal processing chains, adaptations of new classesof applicable physical phenomenon, extensions as to the types and formsof meaningful human control, and, in the context of performance, adeeper integration of visual and audio environments.

SUMMARY OF THE INVENTION

In accordance with some embodiments, the invention provides a signalprocessing and signal synthesis technique from a family of signalprocessing and signal synthesis techniques designed to readily interworkor be used individually in creating new forms of rich musical timbres.Synthetic hysteresis waveshaping is used to create naturally appealingdynamic spectral variations from an input signal of varying amplitudeand waveshape. Waveshaping curves, waveshaping parameters, hysteresischaracteristics and hysteresis parameters may be recalled from storedprogram control or modulated in real-time by arbitrary control signals,including those derived from the original input signal. The inventionmay be used individually or in conjunction with other signal processingand signal synthesis techniques in creating new forms of rich musicaltimbres. The invention may also be used in spatially-distributed timbreconstruction.

Based on research and development of this nature, it is possible tocreate a new-generation framework for expanding the timbral, expressiverange, artistic depth, and semiotic aspects of performed and recordedmusic as well as wide ranges of performance art. Such a framework isparticularly advantageous if it were to build on and inter-work withboth the existing music technology mainstream and the long establishedplaying techniques of expressively sophisticated, iconic, orsignificantly adaptable instruments. With such attributes, isolatedproducts and musical directions can be gently folded in to theestablished mainstream and evolve as the main-stream finds moments ofstagnation and boredom within itself. This methodology would permit thecurrent manufacturing and marketing establishments of music technologyand content to progressively and profitably shift to a more creativelysatisfying and sustainable path.

To these ends, the invention provides methods, apparatus, and exampleimplementations subscribing to a standardized framework which addressthese needs and opportunities.

A key aspect of the invention is a unified architecture involvinginstrument entities, generalized instrument interfaces, and signalrouting, processing, and synthesis elements.

A further aspect of the invention is the defining of general instrumentelements which instrument entities can be created from.

A further aspect of the invention is augmenting existing instrumentlending themselves to expansion with said general instrument elements.

A further aspect of the invention is the use of miniature keyboards forthe attachment to existing instruments.

A further aspect of the invention is the expansion of keyboards toinclude any one or more of proximate, superimposed, programmabletactical feedback, and/or multiple (more than 2) parameter key features.

A further aspect of the invention is the sharing of same electronicsacross multiple keyboards and/or strum-pads.

A further aspect of the invention is that of strum-pads withnon-repeating contacts along the strum path and flexible assignment ofnote event control signals to each contact.

A further aspect of the invention is that of including standardizedarrangements of panel controls, such as switches and sliders, toinstruments.

A further aspect of the invention is the use of null/contact touch-pads,potentially fitted with impact and/or pressure sensors and with thepotential derivation of multiple contact point information, as a musicalinterface.

A further aspect of the invention is that of pressure-sensor arraytouch-pads as an instrument controller, potentially including imagerecognition capabilities and the ability to derive and assign controlparameters from the way the pad is contacted.

A further aspect of the invention is the structuring of associated imageprocessing for a pressure-sensor array touch-pad to capture hand andfoot contact postures and gestures.

A further aspect of the invention is the structuring of associated imageprocessing for a pressure-sensor array touch-pad to derive parametersfrom hand and foot contact postures which permit the application ofuseful metaphors in their operation.

A further aspect of the invention is the implementation ofpressure-sensor array touch-pads, and potentially related decentralizedimage processing and networking functions, in a mini-array chip whichcan be tiled into arbitrary shapes, potentially including instrumentkeys.

A further aspect of the invention is using key displacement togetherwith contact position to derive at least three parameters from astandard Western keyboard key.

A further aspect of the invention is a foot controller with buttons andpedals that have associated alphanumeric displays.

A further aspect of the invention is a foot controller with any one ormore of: hierarchical organization of changeable stored programelements, arbitrary button assignment of hierarchy control functions,and/or multiple interpretation geometric layout of buttons and pedals.

A further aspect of the invention is a method for doing one handed drumrolls with acoustic drums or multiple parameter electronic drumpads.

A further aspect of the invention is: mallets, beaters, and bows withany one or more of impact, grip, position, or pressure, strain, and/ormotion sensors.

A further aspect of the invention is an autoharp adaptation with bothstrings and strum-pads.

A further aspect of the invention is: a string autoharp adaptation wherechord buttons issue control signals.

A further aspect of the invention is an autoharp adaptation where anote-oriented keyboard is used to replaced multiple note chord buttons,potentially where the keys are multiple parameter keys.

A further aspect of the invention is: autoharp, Pipa, Koto, Harp, Mbira,pedal steel, and Sitar adaptations with separate pickups for eachvibrating element, potentially also employing pitch shifting on selectedvibrating element.

A further aspect of the invention is: Pipa, Koto, Harp, Mbira, pedalsteel, and Sitar adaptations with strum-pads.

A further aspect of the invention is: guitar, Pipa, Koto, Harp, Mbira,pedal steel, and Sitar adaptations with vibrating element excitationdrivers built into the instrument.

A further aspect of the invention is: guitar, Pipa, Koto, Harp, Mbira,pedal steel, and Sitar adaptations with additional string arrays and/orone or more miniature keyboards with keys close to the string array.

A further aspect of the invention is the use of vowel synthesis inconjunction with a bowed instrument.

A further aspect of the invention is attaching a video camera to aninstrument.

A further aspect of the invention is the use of optical pickups formetalaphones and drum heads.

A further aspect of the invention is the use of non-equilibrium chemicalreactions as musical controllers or parts of instruments.

A further aspect of the invention is the use of photoacoustic phenomenaas musical controllers or parts of instruments.

A further aspect of the invention is the use of video cameras as musicalcontrollers and/or instruments.

A further aspect of the invention is a wide variety of new signalprocessing innovations, including spatial timbre construction,hysteretic waveshaping, layered signal processing, location modulationof signal pan constellations, cross-product octave chains.

A further aspect of the invention is the provision for a wide variety ofcontrol signal monoatic and polyadic operations as listed in thedisclosure.

A further aspect of the invention is the provision for a wide variety ofcontrol routing capabilities as listed in the disclosure, includingrouting at MIDI message index levels.

The system and method herein can be applied to live performance (music,dance, theater, performance works, etc.), recorded audio and videoproduction, and composition.

DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and advantages of the presentinvention will become more apparent upon consideration of the followingdescription of preferred embodiments taken in conjunction with theaccompanying drawing figures, wherein:

FIG. 1 shows a general overview of the invention;

FIG. 2 shows examples of internal interconnections among the functionalgrouping elements within an instance of a signal routing, processing,and synthesis entity, all shown in FIG. 1;

FIGS. 3A-3C shows an example of a proximate keyboard array;

FIG. 4 shows an example of an instrument-mounted miniature keyboardconfiguration employing one miniature keyboard, in particular an adaptedIndian sitar with many additional example instrument elements;

FIG. 5 shows an example of an instrument-mounted miniature keyboardconfiguration employing two miniature keyboards, in particular anadapted electric guitar with many additional example instrumentelements;

FIGS. 6A-6B-illustrates an arrangement where a dedicated continuous ornear-continuous sensor is attached to each key so as to instantaneouslymeasure the displacement of the attached key;

FIGS. 7A-7B illustrates an arrangement by which programmable tactilefeedback can be applied to a key, either in conjunction or without acontinuous or near-continuous sensor to measure key displacement;

FIG. 8 illustrates a shared scanning arrangement supporting a pluralityof any of keyboards, strum-pad, buttons, switches, etc.;

FIG. 9 illustrates an example method for realizing a flexiblegeneralized strum-pad element and associated stored program control;

FIG. 10 shows an example implementation of both generalized and specificcontrol signals derived from panel controls, actuators, and sensorsusing MIDI;

FIG. 11 shows an example of how two independent contact points can beindependently discerned, or the dimensional-width of a single contactpoint can be discerned, for a resistance null/contact controller with asingle conductive contact plate or wire and one or more resistiveelements whose resistance per unit length is a fixed constant througheach resistive element;

FIG. 12 shows an example implementation of both generalized and specificcontrol signals derived from electrical contact touch-pads employingMIDI messages as the output control signal format;

FIG. 13 shows how a pressure-sensor array touch-pad can be combined withimage processing to assign parameterized interpretations to measuredpressure gradients and output those parameters as control signals;

FIG. 14 illustrates the positioning and networking of pressure sensingand processing “mini-array” chips in both larger contiguous structuresand in isolated use on instrument keys, instrument fingerboards, andinstrument bodies;

FIG. 15 illustrates the pressure profiles for a number of example handcontacts with a pressure-sensor array;

FIG. 16 illustrates how six degrees of freedom can be recovered from thecontact of a single finger;

FIG. 17 illustrates the regions of vowel sounds associated withparticular resonant frequency combinations in vowel sound production;

FIG. 18 illustrates an example two-dimensional timbre space intraditional instrument orchestration;

FIG. 19 shows an example of keys from a traditional Western keyboardfitted with multiple uniformly-sized pressure-sensing and processing“mini-array” chips;

FIG. 20 shows electromagnetic, Hall-effect, piezo, and optical pickupmethods for deriving separate audio signals for each vibrating elementof a multiple vibrating element instrument entity;

FIG. 21 shows how an off-bridge buzz-plate can be combined with a piezobridge sensor in replacement of a gradient buzz-bridge so as to permitthe use of non-ferromagnetic strings;

FIG. 22 shows the basic idea of controlled feedback as used in recentcontemporary music;

FIG. 23 shows an example implementation of a simple approach forreplacing acoustic excitation of a vibrating element withelectromagnetic excitation;

FIG. 24 shows various combinations of piezo and electromagneticvibrating element pickups and exciters for separately controllableexcitation of each vibrating element;

FIG. 25 shows adding signal processing for spectral and amplitudecontrol of electromagnetic excitation;

FIG. 26 shows multiple vibrational elements with common electromagneticexcitation;

FIG. 27 illustrates examples of single, double, and quadruple touch-padinstruments with pads of various sizes and supplemental instrumentelements;

FIG. 28 illustrates some enhanced foot-pedal arrangements which permitsimultaneous single-foot adjustment of a plurality of continuous rangeparameters for use with floor controllers;

FIG. 29 shows some example layouts involving 2 geometric regions for amoderate number of foot operated controllers and 4 geometric regions fora larger number of foot operated controllers;

FIG. 30 shows an example large-scale arrangement of two impact sensorsand/or touch pads for executing one-handed drum-rolls and deriving largeamounts of control information;

FIG. 31 shows an example of an enhanced autoharp implementation asprovided for in the invention;

FIG. 32 shows how the autoharp arrangement of FIG. 31 can be adjusted toreplace its chord button array and associated strum-pads with a keyboardand one or more strum-pads positioned over the keyboard;

FIG. 33 shows an example Koto implementation provided for in accordancewith the invention;

FIG. 34 shows an example Mbira implementation provided for in accordancewith the invention;

FIG. 35 shows an example electric guitar implementation in accordancewith the invention based on a Gibson model ES-335 guitar; theinvention's enhancements shown can be added on as modules, addedcollectively, or built-in;

FIG. 36 shows an example of an adapted European arch-lute with a mix ofsingle strings and double string pairs;

FIG. 37 shows an example pedal steel guitar adaptation as provided forby the invention;

FIG. 38 shows an example flat-necked instrument with a five stringsection emulating a sitar string arrangement and several additionalstrings used for bass or other accompaniment;

FIG. 39 shows an example multiple-pitch sympathetic/buzz/twang resonatorusing banks of short audio delays with high resonances tuned to eachselected pitch, each followed by a dedicated low-speed sweeping flangerwith moderate resonance, a dedicated low-speed sweeping flanger withmoderate resonance, and a low-speed auto-panner;

FIG. 40 shows an example adapted Chinese Pipa as provided for by theinvention featuring a keyboard, strum-pad, touch-pad, slider array,switch array, and impact sensors;

FIG. 41 shows another example adapted Chinese Pipa as provided for bythe invention featuring a bass string array, a harp string array, andimpact sensors;

FIG. 42 shows a bow fitted with sensors to gather information from thehand, bow hairs, and bow motion;

FIG. 43 shows adaptations of a flute and recorder layout with pressuresensors replacing key sites, air turbulence measurements, and airpressure average measurements as provided for in the invention;

FIG. 44 shows how an optical pickup may be created for a suspended gong;this technique may also be used for many other types of metallophones;

FIG. 45 shows example gong arrays as part of a one-hand or two-handpercussion instrument stand;

FIG. 46 illustrates spatial arrays of electrodes which may be used formeasurement, as well as control, in two-dimensional andthree-dimensional configurations;

FIG. 47 shows an arrangement where evolving chemical patterns in thedish of FIG. 46 are illuminated with light sources and visuallymonitored by an overhead camera for any one or more of controlextraction, visual display, or visual recording;

FIG. 48 illustrates example optical measurements of photoacousticphenomena in applicable materials which may be converted to electricalsignals and an example electro-acoustic measurement of photo-inducedacoustic phenomena in applicable materials;

FIG. 49 shows how generalized interfaces can be built in whole or viaseparable parts which may be used selectively as needed or appropriate;

FIG. 50 shows multiple vibrational elements with multi-channeltransducers applied directly to stereo or multi-channel mix-down;

FIG. 51 shows multiple vibrational elements with multi-channeltransducers and individual signal processing prior to mixing;

FIG. 52 shows addition of a control signal extraction element to thearrangement of FIG. 51;

FIG. 53 shows partial mix-downs of vibrating element signals fed to anumber of signal processors and straight-through paths in route tosubsequent mix-down;

FIG. 54 shows a switch matrix assigning signals from vibrating elementsto a number of signal processors en route to subsequent mix-down;

FIG. 55 shows a more flexible method for providing signal processorswith vibrating element signals and other signal processor outputs viaswitch matrix, and additional partial mix-downs by replacing said switchmatrix with a mixer;

FIG. 56 shows configuration control of signal processors, mixers, andswitch matrix, and synthesizer interfaces via logic circuitry and/ormicroprocessing;

FIG. 57 shows a very general combined environment for multi-channelsignal processing, mixing, excitation, and program control of overallconfiguration;

FIG. 58 shows a stereo-input, stereo output configuration of twomonaural flange and/or chorus elements wherein the unaltered signal ofeach input channel is combined with a delay-modulated signal from theopposite channel;

FIG. 59 illustrates a combination of a spatialized effect, twodistortion elements, and a stereo (N=M=2) cross-channel modulated delay;

FIG. 60 illustrates examples of inhomogeneous layered signal processingwhich may be used as shown, with selected omissions, or as an archetypefor similar constructions;

FIG. 61 illustrates an example of a generalized hysteresis modelconstruction as provided for by the invention;

FIG. 62 shows an example implementation of a cross-product octave chainparticularly suited to low cost implementation with logic chips orsimple DSP program loops;

FIG. 63 illustrates an example flexible control and configurationhierarchy for control signal and stored program handling andorganization;

FIG. 64 shows an example method for the generation of control signalsfrom fundamental and overtone information in a signal from a vibratingelement of fixed known pitch;

FIG. 65 shows combining and/or processing fundamental and overtoneinformation obtained from a vibrating element signal prior to parameterextraction;

FIG. 66 shows an example implementation of an adaptive method fortracking overtones for a variable-pitch vibrating element with knownovertone series;

FIG. 67 illustrates an example approach wherein a plurality of LFOs withfeatures as prescribed by the invention may be implemented;

FIG. 68 illustrates traditional stage lighting elements includingover-heads, far-throw, foot, back, floor;

FIG. 69 illustrates example instrument lighting;

FIG. 70 illustrates example rotating speaker emulation light sculptures;

FIG. 71 illustrates light pyramid arrays and light columns arrays; aid

FIG. 72 illustrate stage video projection arrangements.

DETAILED DESCRIPTION

1 Overview

The invention relates to a collection of instruments (adapted,electronic, or combined), generalized instrument electrical interfaces,control signal extraction and manipulation systems, musical synthesismodules, layered audio signal processing, lighting control, lightsculptures, instrument lighting effects, video control, and videodisplay. The resulting rich sonic and visual environment can be used forlive performance, recorded audio and video production, and composition.

FIG. 1 shows a general overview of the invention which, at its highestlevel, consists of one or more instances of instrument entities 100,generalized interface entities 110, and signal routing, processing, andsynthesis entities 120. It is understood that the invention provides forthe possibility of several instances of each of these entities. Forexample, several instruments 100 may be supported (adapted guitar,adapted sitar, adapted autoharp, touch-pad/slider/switch controller,etc.) by each 110/120 system; further, for each collection of pluralityof instruments 100 and signal routing, processing, and synthesisentities 120, there may be both full-feature interface cables orsimplified reduced-feature cables implementing versions of thegeneralized interface 110; finally, for each collection of instruments100 and generalized interface entities 110 there may be various versionsof signal routing, processing, and synthesis entities 120 (smallperformance systems, large performance systems, studiorecording-oriented or composition-oriented systems, etc.).

1.1 Instrument Overview

In more detail, each instrument entity 100 in general internallyconsists of one or more elements. The elements fall into two broadcategories, namely those that produce audio-frequency signals and thosethat instead produce only control signals. Of these, it is also possibleto derive control signals from the audio-frequency signals (reflectingpitch, amplitude, relative harmonic content, etc.). Control signals,regardless of their origin, in general are used to control theprocessing, replay, or synthesis of audio-frequency signals; however,the control signals can also be used to control lighting, video, specialeffects, etc.

Referring to FIG. 1, the instrument 100 may contain internal powersources (such as batteries, large-value capacitors, etc.) and/or powerregulation elements 101. Next the control signal sources that may beincluded within an instrument entity 100 may be of a traditionaltechnology or nature, such as knobs, keys, switches, touch-pads,sliders, buttons, sensors, etc.; these will be termed electronicinterface elements 102. In addition, it is also possible to generatecontrol signals from more exotic processes such as chemical oscillators,chemical chaos, photoacoustic, environmental sensors, etc. These will betermed alternative control signal elements 103. The audio-frequencysignal sources that may be included within an instrument also broadlyfall into two classes. One class is that of traditional vibratingelements (strings, tynes, surfaces, solid volumes, air columns, etc.)whose mechanical audio-frequency vibrations can be electrically sensedvia electromagnetic, photo-electric, piezo, Hall-effect, or other typesof sensors or transducers. In many cases it is possible to excite thesemechanically vibrating elements by electronic methods (magnetic fields,piezo transducers, etc.) or electronically controlled methods (motorizedbowing, solenoid strikers, etc.) This first class of audio-frequencysignals sources will be termed sensed/excited vibrating elements 104. Inaddition, it is also possible to generate audio-frequency signals frommore exotic processes such as chemical oscillators, chemical chaos,photoacoustic, environmental sensors, etc. These will be termedalternative control signal elements 105. Finally, the instrument mayalso contain various additional video, lighting, and special effectelements 106.

1.2 Generalized Interface Overview

Again referring to FIG. 1, the invention provides for both instrumententities 100 and signal routing, processing, and synthesis entities 120to be fitted with compatible electrical interfaces, termed generalizedinstrument interfaces or (or more concisely, generalized interfaces)110, which can exchange any of the following:

-   -   incoming electrical power (111)    -   outgoing control signals from switches, controls, keyboards,        sensors, etc., typically in the form of MIDI messages but which        may also involve contact closure or other formats (112)    -   control signals to lights, pyrotechnics, or other special effect        elements within and/or attached to the instruments, said signals        being either in the form of MIDI messages, contact closure, or        other formats (113)    -   outgoing audio signals from individual audio-frequency elements        or groups of audio-frequency elements within the instruments        (114)    -   incoming excitation signals directed to individual        audio-frequency elements or groups of audio-frequency elements        within the instruments (115)    -   outgoing video signals (such as NTSC, PAL, SECAM) or image        signals sent from the instrument (116)    -   incoming video signals (such as NTSC, PAL, SECAM) or image        signals sent to the instrument for purposes such as display or        as part of a visually controlled instrument (117).

The interfaces may be realized by one or more of any of connectors,cables, fibers, radio links, wireless optical links, etc.

1.3 Signal Routing, Processing, and Synthesis Overview

Referring to FIG. 1, the invention provides for one or more signalrouting, processing, and synthesis entities 120. These entities firstroute and process received audio-frequency, control, and video signals.Additionally, these entities 120 may extract control signals fromreceived audio-frequency and video signals, perhaps under the directionof selected control signals. Finally, these entities 120 may alsosynthesize audio-frequency, control, and video signals, typically underthe direction of selected control signals.

Again referring to FIG. 1, the signal routing, processing, and synthesisentities 120 internally may include:

-   -   power supplies 121 for both internal and instrument powering    -   control signal routing 122 for interconnecting control signal        sources with control signal destinations    -   control signal processing 123 for instantaneous control message        transformations (such as inversions) and inter-operations (such        as averaging, adding, multiplication, etc.) audio signal routing        124 for interconnecting audio signal sources with audio signal        destinations    -   audio signal processing 125 for (typically real-time)        transformations, typically under real-time control via selected        control signals    -   video signal routing 126 for interconnecting video signal        sources with audio signal destinations, typically under        real-time control via selected control signals video signal        processing 127 for (typically real-time) video signal        transformations, potentially under real-time control via        selected control signals    -   control signal extraction 128 a for the derivation of (typically        real-time) control signals from audio or video signals,        potentially under real-time control via selected control signals    -   control signal synthesis 128 b for the internal creation of        time-varying control signals (such as low-frequency control        oscillators, envelop generators, slew limiters, etc.),        potentially under real-time control via selected control signals    -   audio signal synthesis 129 a, typically under the direction of        selected control signals, and typically as per conventional        music synthesizer hardware and software    -   video signal synthesis 129 b, typically under the direction of        selected control signals.    -   program storage 130 for storing configuration programs and event        sequences

In FIG. 1 it is understood that the elements 121 through 130 representfunctional groupings and not necessarily hardware-centralized orsoftware-centralized sub-systems.

FIG. 2 shows examples of internal interconnections among the functionalgrouping elements 121 through 130 within an instance of a signalrouting, processing, and synthesis entity 120. In FIG. 2, as before, itis understood that the elements 121 through 130 represent functionalgroupings and not necessarily hardware-centralized orsoftware-centralized subsystems.

In the example interconnections, power is distributed throughout viafunctional fan-outs 131; here it is understood that there many be manydecentralized power sup-plies for the individual subsystems comprisingor implementing elements 122-130. Program store information is alsodistributed throughout via paths 132 (associated with specificsubsystems of elements 122-129) and/or path 133 to the control signalrouting element 122; typically both methods are used as portions of theprogram control may be stored within individual elements 122-129 andportions may reside within one or more centralized program storesubsystems (such as MidiTemp model MP-88, Digital Music Corporationmodel MX-8, controlling PC, etc.), comprising 130.

1.4 Remaining Document Overview

With this overview complete, the remainder of the discussion isorganized as follows. The next four Sections concern instruments 100.First, a number of instrument element and instrument subsystems aredescribed. Two subsequent sections then describe a large number ofexample instruments that are perfected through applicable combinationsand arrangements of the aforementioned instrument elements andsubsystems of elements; the first of these sections purely electroniccontrollers while the second addresses adaptations of conventionalinstruments with special attention paid to specific nuances andopportunities within those instruments. Following this, some alternativeaudio and control signal sources are then considered.

Next the general instrument interface 110 is then considered inadditional detail. A subsequent section then addresses the signalprocessing, and synthesis entities 120. A final section provides a fewexample envisioned applications of the invention.

2 Instrument Elements and Instrument Subsystems

The invention includes a number of electronically interfaced instrumentsused by one or more performers.

These instruments involve either pure electronic interfaces arranged toform an instrument, vibrating elements which typically are inarrangements adapted from existing instruments, exoticelectrically-monitored oscillatory elements (such as chemicaloscillators), electronic or numerical chaotic models used as sources, orcombinations of these laid out in an artistically operative andergonomic fashion. Vibrating elements within an instrument may also bemade to vibrate via electronically controlled or induced excitation frommagnetic field, piezo electromechanical, or other electronically-drivenor electronically-controlled excitation.

In general an instrument consists of one or more instrument elementswhich may be of one more differing types or classes. These instrumentelements may be thought of as subsystems within the instrument. Forexample, a 6-string guitar has six vibrating strings; each string is anexample of a vibrating element. A single electromagnetic or piezo pickupmay be used to amplify the entire group of six strings. The guitar mayalso have separate electromagnetic or piezo pickups for each string, asis commonly done for adding a MIDI interface to an existing electricguitar. This example guitar then simultaneously has six vibratingelements, one group-pickup subsystem, and six single-string pickupsubsystems. The guitar may be further enhanced with MIDI-command issuingcontrols, such as knobs, switches, joysticks, touch-pads,motion/position sensors, etc.; these represent an additional subsystem.A reduced-size musical keyboard may be added to the guitar, representingyet another subsystem.

Specific classes of instrument elements and/or instrument subsystems aredescribed in the subsections that follow.

2.1 Electronic Interface Instrument Elements and Subsystems

This class of instrument elements and instrument subsystems do notcreate audio frequency phenomenon directly but are rather used tocontrol one or more music synthesizers, audio mixers, and/or signalprocessing functions.

2.1.1 Proximate, Miniature, and Superimposed Keyboards

Standard western keyboards found on pianos, harpsichords, organs, andsynthesizers are widely used as a human interface for electronic musicalinstruments. Some instruments, such as organs and harpsichords, havetraditionally (for centuries) included two or more such keyboards toallow the instrument player to rapidly select among two or more timbresor ranges. The spacing of the keyboards is almost without exceptionfound to be far enough apart that a hand must be committed uniquely to agiven keyboard for the moments that the keys are played. This is due tothe fact that the bulk of apparatus under the keyboards, keyboard frame,etc. prevented the keyboards from being mounted very close together,re-enforced by the fact that music has been composed for playing at mostone keyboard with a given hand (although in virtuoso pieces a given handmay very rapidly jump among keyboards). One aspect of the inventionexpands the usage of traditional keyboards by removing this limitationvia various means.

2.1.1.1 Proximate Keyboard Arrays

One method of implementation is to mount a plurality of keyboards closeenough together that one hand can, to degrees determined by mechanicaldetails, simultaneously play notes on two or more traditional keyboards.There are three methods for increasing the workable proximity of groupsof keyboards:

-   -   reduce the vertical separation of the keyboards    -   overhang the ends of the white keys on a higher keyboard over        the backs of the white and black keys of a lower keyboard    -   reduce the physical length of the keys

Many modern electronic keyboards have very shallow mechanisms andframes. It is therefore quite straightforward to mount two or morecommonly available electronic keyboards employing either or both of thefirst two methods. With some overhang and (vertically or horizontally)shallow enough mechanisms, it becomes possible to play notes on bothkeyboards simultaneously. In nominal configurations the thumb-to-pinkyreach is nearly the same across both keyboards. Clearly some fingerconfigurations are difficult or impossible across the two keyboards, butthere are also limitations in conventional keyboards that areincorporated in the development of established fingering technique andrespected in keyboard music composition; similar minor techniquedevelopment and compositional respect extensions can be developed forsuch proximate keyboard arrays.

Without reducing the size of the keyboards a single hand can even makeinvaluable use of three keyboards within a confined range; simpleexample is to add back-up notes of the same pitch or differing octaves.However, two hands may use the two-keyboard playing techniques to makeavid use of a three, four, or more proximate keyboard array.

FIGS. 3A-3C shows an example of a proximate keyboard array. In thisexample, three keyboards 301, 302, 303 are arranged in an overhangingstaircase arrangement. Three views are shown: a side view 300 (FIG. 3A),a top view with hidden key areas suggested by dashed lines 310 a (FIG.3B), and the side view of 300 reoriented as an orthogonal projection 310b of the top view 310 a (FIG. 3C). The separation distances 305 a, 305 bbetween the tops of the keys of a given keyboard and the bottoms of thekeys of a keyboard overhanging it should be minimized and in the limitare just slightly larger than the maximum travel distance of theoverhanging key. The depth of the overhang 304 a, 304 b is set in thebalance between the trade-off of maximizing desired accessibility to theback of an overhung key and minimizing the separation distance betweenthe edges of the keys of two adjacent proximate keyboards. It is notedthat any of the keyboards used here may be either of a standard varietyor any of the more advanced keyboards described later (miniature,superimposed, multi-parameter keys, pressure-sensor array, etc.). It isalso noted that this technique may be applied to other types ofkeyboards with applicable types of key geometry.

2.1.1.2 Miniature Keyboards

If the depth of the keyboard is reduced, the span of a given hand isincreased further. This may be done by making the keys relativelyshorter, forming a stubby keyboard, or by shrinking the size of theentire keyboard in all dimensions. Such miniature keyboards are commonlyfound on consumer electronic keyboards and keyboard instruments made forchildren.

Clearly a proximate keyboard array can be created from miniaturizedkeyboards. The range of the fingers within and across individualcomponent keyboards may be greatly increased in this fashion, albeitwith a perhaps somewhat compromised tradition and technique.

An additional, and particularly valuable role for the proximatecapabilities of such miniature keyboards is to mount them, as acomponent, on an instrument with other components so as to form a morecomplex instrument where free fingers can operate two or more suchcomponents simultaneously. As a simple example, a guitarist using athumb-pick or classical guitar technique can easily use free fingers toplay chords, bass lines, melodies, etc. on a miniature keyboard attachedto a guitar.

FIG. 4 shows an example of an instrument-mounted miniature keyboardconfiguration employing one miniature keyboard 421, in particular anadapted Indian Sitar 400 with many additional example instrumentelements which will be described later.

FIG. 5 shows an example of an instrument-mounted miniature keyboardconfiguration employing two miniature keyboards, in particular anadapted electric guitar 500 with many additional example instrumentelements which will be described later. Here note the two keyboards 521a, 521 b are proximate enough to allow both keyboards and the guitarstrings to be played simultaneously.

Clearly these methods of miniature keyboard attachment(s) can be appliedto other instruments (Sitar, Pipa, Saz, pedal steel guitar, pluckedstring bass, etc.) as well as being used to create entirely new types ofinstruments and controllers as will be discussed herein.

2.1.1.3 Superimposed Keyboards

It is also possible to make contact-closure keyboards with multiplecontact sets that actuate at increasing depths of key depression. Suchkeyboards may or may not have tactile feedback as to each level ofactuation. Pratt-Read manufactured a “double-touch” keyboard for use inhome console organs which closed one set of contacts with a noticeablerestoring pressure at about half of the possible key-displacement whichpersisted through full key displacement where another contact set closedat the end of key travel. Also, many “velocity sense” keyboards arerealized by SPDT switches actuated with each key; here the beginning ofkey travel opens a pair of contacts and the end of key travel closes asecond set of contacts, but with no mid-travel tactile feedback.

In either case, there are one contact closure event at partial keytravel and two events at full key travel. These events can beinterpreted as superimposed keyboards. One example interpretation isthat the first event triggers one synthesizer voice and the second eventriggers a second voice; in this manner keys struck with partialdisplacement sound with only one voice but those struck with fulldisplacement sound both voices. Another example is for a first voice tobe triggered at partial displacement but turned off at fulldisplacement. If the first voice has a long attack, it would be drownedout by the second voice, or in short duration serve as acceptabletransient ornamentation (for example, mimicking a “key click” or “airturbulence chiff”), this arrangement effectively resulting in a partialkey displacement sounding only the first voice and a full keydisplacement sounding only the second voice. Note in either arrangement,a fluctuation of the applied key pressure can vary which voices continueto sound (in the first arrangement, the second voice will go on and offwith the first voice held; in the second arrangement, the first andsecond voices will alternate being on or off in a mutually exclusivefashion).

As the superposition of keyboard principal proves useful in thistwo-level setting, it is natural to consider further extensions of thisapproach to more levels and additional interpretations. In the limit, akeyboard could have a continuous sensor (such as a potentiometer,magnetic or optical gradient, etc.) or near-continuous sensor (such as abinary encoded control) attached to each key. FIGS. 6A-B illustrates anarrangement where a dedicated continuous or near-continuous sensor isattached to each key so as to instantaneously measure the displacementof the attached key. In such an arrangement external electronics woulddefine quantized displacement thresholds to which various superimposedkeyboard interpretations would be assigned.

As a first bonus, it is also noted that this same continuous ornear-continuous key-displacement sensor arrangement can be used in otheroperational modes to provide other very valuable expressive functions,for example volume or timber control or velocity contour tracking, aswill as will be described in a later section.

In practice, the two-level superimposed keyboard provides the playerwith tactile feedback as to what point of travel the key had passed inthe form of a noticeable change in resistive restoring pressure. For amore generalized system as described above and illustrated in FIGS.6A-6B, there may be applications where such tactile feedback is notespecially necessary, for example in triggering additional synthesizervoices to create an increasing gradient of richness as the key ispressed further and further. In other circumstances, particularly ifthere are only a few levels implemented, tactile feedback may indeed bedesirable, particularly that with discernible discrete steps matchingthe trigger-level quantization points in key travel.

Highly flexible programmable tactile feedback can be imposed separatelyon each key by a dedicated solenoid, motor, pneumatic, fluid, or othermeans. Less flexible yet still somewhat programmable tactile feedbackcould also be had by means of an electrically adjustable globalmechanical arrangement serving all keys in a keyboard, for exampleengaging additional sets of springs or pliable rubber pressure-resistingcones. FIGS. 7A-B illustrates an arrangement by which programmabletactile feedback can be applied to a key, either in conjunction with orwithout a continuous or near-continuous sensor to measure keydisplacement. Without key position information, anelectrically-controlled restoring force element with built-in levels ofkey pressure resistance (for example, by means of a sequence ofspatially distributed electromagnetic coils that can be switched on atconfiguration time to create additional levels of force past specificdisplacement depths) could be used. With key displacement information, asimple dedicated solenoid, motor, pneumatic, fluid, or other means canbe made to have its restoring force vary over the key travel in a highlyflexible manner. Since key travel can be fast, the transient response ofthe tactile feedback system must typically have a fast rise time and befree of overshoot. If electromagnetic or electric field means are usedto provide key displacement resistance, care must be made to shieldthese elements to as to not create electromagnetic transients that couldleak into nearby electronics or music instrument pickups.

Finally, it is pointed out that as an additional bonus, the abovearrangement is also capable of synthesizing different types ofmechanical so-called keyboard “actions”, for example the “feel” ofvarious types of piano manufacture keys versus harpsichord keys, etc.Thus the development of a keyboard with per-key continuous ornear-continuous displacement measurements and programmablekey-displacement resistance can provide an extraordinary level ofenhancements to conventional keyboards. This can be enhancedsignificantly with the addition of pressure sensing arrays on each keyas will be described later.

2.1.1.4 Shared Scanning Electronics

In arrangements with multiple keyboards, superimposed keyboards, orrelated input devices (such as the strum-pads discussed below) thekeyboard-scanning electronic hardware can be in many cases largelyshared across pluralities of these keyboard contacts and/or relatedinput devices. For example, a common microprocessor could be used togenerate common multiplexing address for a group of contacts or sensorsacross several keyboards and the status of individual contacts wouldthen be serially polled or transferred in parallel. FIG. 8 illustrates ashared scanning arrangement supporting a plurality of any of keyboards,strum-pad, buttons, switches, etc.

2.1.2 Strum-Pads

A few early music synthesizers replaced a conventional keyboard with alow-activation pressure membrane switch array laid out to resemble akeyboard. One could freely tap or easily drag fingers over the membraneswitch array without the overhead and potential injury involved in moredeeply operative conventional keyboards. Because of the lack ofconventional keyboard action and technique, such keyboards rapidly losttheir appeal. More recently, the Suzuki “Omnichord” product, designed tomimic an autoharp, provided a low-activation pressure membrane switcharray, called a “strum-pad,” laid out to mimic the strummed-string arrayof an autoharp; as a selected chord button is activated various notesassociated with the chord are assigned to the various membrane switchesso that a finger sweeping over the strum-pad produces an arpeggiatedchord in a way suggestive of strumming a traditional autoharp. TheOmnichord strum-pads are hard-wired to repeat notes multiple times andthe note assignment software permits only fixed chord selections withpreassigned arpeggio note sequences.

The invention includes an important element to create or expandinstruments through a generalized adaptation of these ideas:

-   -   a more generalized strum-pad element with the following        attributes:    -   low activation-pressure proximate switches    -   linear arrangement (although others are useful)    -   no hard-wired note repeats    -   visual and/or small tactile markings to the player    -   compact physical size    -   simultaneous multiple switch activation without perceivable        interaction    -   generalized note event information that can be assigned        interpretation under program control    -   more generalized strum-pad interpretation software and hardware        with the following stored program attributes and assignments        which can be rapidly altered during playing:    -   assignment to selected melodic notes, percussive events,        lighting or special effect events, etc.    -   arpeggio pattern select    -   note-repeats added as desired and in the manner desired    -   issuance of note, outgoing program change, and/or other control        signals at the initial activation of each stored program (to        sound a background chord, activate lights, etc.) with or without        activity on the strum-pad    -   selection and rapid change of specific programmable attributes        and assignments via button or foot-switch control.

The resulting element can, for example, be attached to a guitarpick-guard and used in conjunction with foot-switches and/orfinger-activated buttons to select stored program interpretations. Freefingers can then, while freely playing the guitar as normal, “strum” ortap arpeggios, trigger percussion devices, trigger lighting or specialeffect events, etc.

FIG. 9 illustrates an example method for realizing a flexiblegeneralized strum-pad element and associated stored program control. Inthis example implementation, the strum-pad switches can be electricallywired to a simple conventional MIDI keyboard interface so that eachconsecutive switch triggers a consecutive MIDI note event. The noteevent stream is then directed to a MIDI message processor which can,under program control, reassign each incoming note event a potentiallynew MIDI note number and MIDI channel, or perhaps a null operation tocreate “safety” or “dead” zones. From here individual MIDI channels canbe directed to a variety of destinations: various synthesizer voicechannels, lighting systems, special effect systems, etc. Additionalcontrol possibilities can be further realized by translating note eventsinto other types of MIDI events, as described later, or into non-MIDIcontrol signals.

It is also possible to add note-velocity and/or“key-pressure”/“after-touch”/“channel-pressure” control to the strum-padby placing a velocity sensor (such as a piezo element) and/orpressure-sensor under it and feeding the resulting signal(s) to the MIDIkeyboard interface as would be done in a conventional MIDI keyboardrealizing these features with such sensors. It is also possible tosupplement, or replace altogether, each membrane switch with apressure-sensor, thus creating a pressure-sensor array. Such an arraycan be used to implement note-velocity and/or“key-pressure”/“after-touch”/“channel-pressure” control, but can also beused for a great many other purposes, particularly when implemented in atwo-dimensional array, as described later.

2.1.3 Panel Controls, Actuators, Sensors

Expressive control can be enhanced considerably by attaching one or moreof any of various additional panel controls, actuators, and sensors toany electronic instrument.

Applicable types of panel controls include potentiometers (knob, slider,etc.), joysticks, panel switches, panel buttons, etc. Panel controls maybe distributed in isolated spots, in small groups, or in arrays.

Applicable actuators can include limit switches, magnetic switches,mercury switches, optical detectors, piezo or other impact detectors,etc. Actuators may be attached or associated with moveable parts ofinstruments (such as guitar vibrato “whammy” bars, harp tuning levers,autoharp string-damper bars, etc.). Additionally, actuators may beaffiliated with the instrument as a whole, detecting rapid jarring ofthe instrument etc. Further, actuators may also be provided in isolatedspots of the instrument, such as velocity-sensitive tap-actuators forpercussion event-triggers and “body blows” to the instrument, asabstracted from for examples: ancient Chinese Pipa, centuries oldFlamenco guitar, and recent Jimi Hendrix/Adrian Belue (borderline toactual guitar abuse) techniques.

Applicable sensors can include pressure, motion (velocity, acceleration,etc.), position (optical, magnetic or electric field, electromagneticstanding wave, acoustic standing wave, etc.), impact (such as piezosensors used with electronic drum pads), tension, strain, torsion,light, temperature, etc. Position sensors may be used to measure theposition of a physical element of an instrument (such as a damper bar orpitch-modulating lever) or the absolute position of the instrumentitself. Tension sensors may be used, for example, to measure modulatedstring tension as on a Koto or electric guitar; such string tensioncontrollers need not even involve sounding strings—for example a smallKoto string and bridge arrangement may be used strictly as an electroniccontrol provided to the player in the form of a familiar Koto stringformat.

In general these panel controls, actuators, and sensors can beconfigured to provide a range of either continuous or discrete-stepcontrol voltages. In some cases additional electronics or subsequentsoftware transformations may be necessary to re-contour/redistribute thecontrol voltage over the full range of the controls, actuators, and/orsensors. In some cases, multiple transformations may be made availableunder selectable or stored program control. In any case, the resultingcontrol voltages may be then treated as generalized control signalswhich are presented to the generalized interface 110. Alternatively,some of the control voltages may be used for specialized controlsignals, such as setting values for note-velocity, after-touch, etc.

FIG. 10 shows an example implementation of both generalized and specificcontrol signals derived from panel controls 1001.1-1001.n, actuators1002.1-1002.m, and sensors 1003.1-1003.k as provided for by theinvention. The panel controls 1001.1-1001.n, actuators 1002.1-1002.m,and sensors 1003.1-1003.k may or may not be provided with appropriateinterface electronics, respectively 1011.1-1011.n, 1012.1-1012.m, and1013.1-1013.k which deliver signals to a control signal formatter 1050which issues control signals 1051. These control signals may be ofvarious formats, for example MIDI.

2.1.4 Null/Contact Touch-Pads

Distinguished from panel controls and sensors considered above are whatwill be termed null/contact touch pads. This is a class ofcontact-position sensing devices that normally are in a null stateunless touched and produce a control signal when touched whose signalvalue corresponds to typically one unique position on the touch-pad.Internal position sensing mechanisms may be resistive, capacitive,optical, standing wave, etc. Examples of these devices includeone-dimensional-sensing ribbon controllers found on early Musicsynthesizers, two-dimensional-sensing pad such as the early Kawala padand more modern mini-pads found on some lap-top computers, andtwo-dimensional-sensing see-through touch-screens often employed inpublic computer kiosks. As a music controller these devices areattractive in that they can very easily capture very expressive fingernuances as does a violin fingerboard or Koto bridge/string arrangementbut not limit them to controlling only pitch. Two-dimensional versionsof these devices also permit the use of spatial metaphors and notions of“musical finger-painting.”

The null condition, when the pad is untouched, requires and/or providesthe opportunity for special handling. Some example ways to handle theuntouched condition include:

-   -   sample-hold (hold values issued last time sensor was touched, as        does a joystick)    -   bias (issue maximal-range value, minimal-range value, mid-range        value, or other value)    -   touch-detect on another channel (i.e., a separate out-of-band        “gate” channel).        Example uses for these devices as controller elements within the        context of the invention include any one or more of the        following:    -   issuance of melodic or percussion note events    -   pitch, amplitude, timbre, and location (i.e., panning, etc.)        modulations    -   lighting and/or special effects control    -   general MIDI CC control signals.

Additional enhancements can be added to the adaptation of null/contacttouch pad controllers as instrument elements. A first enhancement is, asdiscussed above for strum-pad elements, the addition of velocity and/orpressure sensing. This can be done via global impact and/orpressure-sensors in the same manner as described for the strum-pads. Anextreme of this is implementation of the null/contact touch padcontroller as a pressure-sensor array; this special case and its manypossibilities are described later. On the simpler extreme, anull/contact touch pad together with such a global velocity and/orpressure-sensor can act as a rich metaphor for a drum head, gongsurface, cymbal surface, etc. and as such may be played with fingers,whole hands, cushioned beaters, or sticks.

A second enhancement is the ability to either discern eachdimensional-width of a single contact area or, alternatively,independently discern two independent contact points in certain types ofnull/contact controllers. FIG. 11 shows an example of how twoindependent contact points can be independently discerned, or thedimensional-width of a single contact point can be discerned, for aresistance null/contact controller with a single conductive contactplate (as with the Kawala pad product) or wire (as in a some types ofribbon controller products) and one or more resistive elements whoseresistance per unit length is a fixed constant through each resistiveelement. It is understood that a one-dimensional null/contact touch padtypically has one such resistive element while a two-dimensionalnull/contact touch pad typically has two such resistive elements thatoperate independently in each direction.

Referring to FIG. 11, a constant current source can be applied to theresistive element as a whole, developing a fixed voltage across theentire resistive element. When any portion of the resistive element iscontacted by either a non-trivial contiguous width and/or multiplepoints of contact, part of the resistive element is shorted out, thusreducing the overall width-to-end resistance of the resistance element.Because of the constant current source, the voltage developed across theentire resistive element drops by an amount equal to the portion of theresistance that is shorted out.

The value of the voltage drop then equals a value in proportion to thedistance separating the extremes of the wide and/or multiple contactpoints. By subtracting the actual voltage across the entire resistiveelement from the value this voltage is normally, a control voltageproportional to distance separating the extremes of the wide and/ormultiple contact points is generated. Simultaneously, the voltagedifference between that of the contact plate/wire and that of the end ofthe resistive element closest to an extremal contact point is stillproportional to the distance from said end to said extremal contactpoint. Using at most simple op-amp summing and/or differentialamplifiers, a number of potential control voltages can be derived; forexample one or more of these six continuously-valued signals:

-   -   value of distance difference between extremal contact points (or        “width”; as described above via constant current source, nominal        reference voltage, and differential amplifier)    -   center of a non-trivial-width region (obtained by simple        averaging, i.e., sum with gain of ½)    -   value of distance difference between one end of the resistive        element and the closest extremal contact point (simple        differential amplifier)    -   value of distance difference between the other end of the        resistive element and the either extremal contact point (sum        above voltage with “width” voltage with appropriate sign).

Further, through use of simple threshold comparators, specificthresholds of shorted resistive element can be deemed to be, forexample, any of a single point contact, a recognized contact regionwidth, two points of contact, etc., producing correspondingdiscrete-valued control signals. The detection of a width can be treatedas a contact event for a second parameter analogous to the singlecontact detection event described at the beginning. Some example usageof these various continuous and discrete signals are:

-   -   existence of widths or multiple contact points may be used to        trigger events or timbre changes    -   degree of widths may be used to control degrees of modulation or        timbre changes    -   independent measurement of each extremal contact point from the        same end of the resistive element can be used to independently        control two parameters. In the simplest form, one parameter is        always larger than another; in more complex implementations, the        trajectories of each contact point can be tracked (using a        differentiator and controlled parameter assignment switch); as        long as they never simultaneously touch, either parameter can        vary be larger or smaller than the other.

It is understood that analogous approaches may be applied to othernull/contact touch pad technologies such as capacitive or optical.

A third possible enhancement is that of employing a touch-screeninstance of null/contact touch pad and position it over a video display.In this case the video display signal may be created either within aninstrument entity 100, within the signal routing, processing, andsynthesis entity 120, or from external sources such as stage cameras,attached computers, etc. The video display could for example providedynamically assigned labels, abstract spatial cues, spatial gradients,line-of-site cues for fixed or motor-controlled lighting, etc. whichwould be valuable for use in conjunction with the adapted null/contacttouch pad controller.

These various methods of adapted null/contact touch pad elements can beused stand-alone or arranged in arrays (as in a percussion controller).In addition, they can be used as a component or addendum to instrumentsfeaturing other types of instrument elements. FIG. 12 shows an exampleimplementation of both generalized and specific control signals derivedfrom electrical contact touch-pads employing MIDI messages as the outputcontrol signal format.

2.1.5 Pressure-Sensor Array Touch-Pads

The invention provides for the selective inclusion of considerablyadvanced expressive control of electronic musical processes through useof a pressure-sensor array arranged as a touch-pad together withassociated image processing. As with the null/contact controller, thesepressure-sensor array touch-pads may be used stand-alone, organized intoan array of such pads, and/or used as a component and/or addendum toinstruments employing other types of instrument elements.

It is noted that the inventor's original vision of the below describedpressure-sensor array touch-pad was for applications not only in musicbut also for computer data entry, computer simulation environments, andreal-time machine control, applications to which the below describedpressure-sensor array touch-pad clearly can also apply.

A pressure-sensor array touch-pad of appropriate sensitivity range,appropriate “pixel” resolution, and appropriate physical size is capableof measuring pressure gradients of many parts of the flexibly-rich humanhand or foot simultaneously. FIG. 13 shows how a pressure-sensor arraytouch-pad can be combined with image processing to assign parameterizedinterpretations to measured pressure gradients and output thoseparameters as control signals.

The pressure-sensor “pixels” 1300 of a pressure-sensor array touch-pad1301 are interfaced to a data acquisition stage 1302. The interfacingmethod may be fully parallel but in practice may be advantageouslyscanned at a sufficiently high rate to give good dynamic response torapidly changing human touch gestures. To avoid the need for a bufferamplifier for each pressure-sensor pixel 1300, electrical design maycarefully balance parasitic capacitance of the scanned array with theelectrical characteristics of the sensors and the scan rates; electricalscanning frequencies can be reduced by partitioning the entire arrayinto distinct parts that are scanned in parallel so as to increase thetolerance for address settling times and other limiting processes.Alternatively, the pressure-sensor array 1301 may be fabricated in sucha way that buffer amplifier arrays can be inexpensively attached to thesensor array 1301, or the sensors 1300 may be such that each containsits own buffer amplifier; under these conditions, design restrictions onscanning can be relaxed and operate at higher speeds. Although thepressure-sensors may be likely analog in nature, a further enhancementwould be to use digital-output pressure-sensor elements or sub-arrays. Aparticularly useful example of sensor sub-arrays is presented in a fewparagraphs.

The data acquisition stage 1302 looks for sensor pixel pressuremeasurement values that exceed a low-levelnoise-rejection/deformity-rejection threshold. The sufficiently highpressure value of each such sensor pixel 1300 is noted along with therelative physical location of that pixel (known via the pixel address).This noted information may be stored “raw” for later processing and/ormay be subjected to simple boundary tests and then folded intoappropriate running calculations as will be described below. In general,the pressure values and addresses of sufficiently high pressure valuepixels are presented to a sequence of processing functions which may beperformed on the noted information:

-   -   contiguous regions of sufficiently high pressure values are        defined (a number of simple run-time adjacency tests can be        used; many are known—see for example [Ronse; Viberg; Shaperio;        Hara])    -   the full collection of region boundaries are subjected to        classification tests; in cases a given contiguous region may be        split into a plurality of tangent or co-bordered independently        recognized regions    -   various parameters are derived from each independent region, for        example geometric center, center of pressure, average pressure,        total size, angle-of-rotation-from-reference for non-round        regions, second-order and higher-order geometric moments,        second-order and higher-order pressure moments, etc.    -   assignment of these parameters to the role of specific control        signals (note events, control parameters, etc.) which are then        output to the signal routing, processing, and synthesis entity        120; for example, this may be done in the form of MIDI messages.

Because of the number processes involved in such a pipeline, it isadvantageous to follow a data acquisition stage 1302 with one or moreadditional processing stages 1303. Of the four example processingfunctions listed above, the first three fall in the character of imageprocessing. It is also possible to do a considerable amount of the imageprocessing steps actually within the data acquisition step, namely anyof simple adjacency tests and folding selected address and pressuremeasurement information into running sums or other runningpre-calculations later used to derive aforementioned parameters. Thelatter method can be greatly advantageous as it can significantlycollapses the amount of data to be stored.

Regardless of whether portions of the image processing are done withinor beyond the data acquisition stage, there are various hardwareimplementations possible. One hardware approach would involve verysimple front-end scanned data acquisition hardware and a singlehigh-throughput microprocessor/signal-processor chip. Alternatively, anexpanded data acquisition stage may be implemented in high-performancededicated function hardware and this would be connected to a lowerperformance processor chip. A third, particularly advantageousimplementation would be to implement a small pressure-sensor arraytogether with data equitation and a small processor into a singlelow-profile chip package that can be laid as tiles in a nearly seamlesslarger array. Such “mini-array” chips have additional value as they canreadily be put on instrument keys (as described below), instrumentfingerboards, instrument bodies, etc. In such an implementation allimage processing could in fact be done via straightforward partitionsinto message-passing distributed algorithms.

One or more individual chips could direct output parameter streams to anoutput processor which would organize and/or assign parameters to outputcontrol channels, perhaps in MIDI format, perhaps in a programmablemanner under selectable stored program control. A tiled macro array ofsuch “sensor mini-array” chips could be networks by a tapped passivebus, one- or two-dimensional mode active bus daisy-chain, a potentiallyexpandable star-wired centralized message passing chip or subsystem, orother means.

Creating a large surface from such “tile chips” will aid in theserviceability of the surface. Since these chips can be used as tiles tobuild a variety of shapes, it is therefore possible to leverage asignificant manufacturing economy-of-scale so as to minimize cost andjustify more extensive feature development. Advanced seating andconnector technologies, as used in lap-tops and other high-performanceminiature consumer electronics, can be used to minimize the separationbetween adjacent chip “tiles” and resultant irregularities in thetiled-surface smoothness. A tiled implementation may also include a thinrugged flexible protective film that separates the sensor chips from theoutside world. FIG. 14 illustrates the positioning and networking ofpressure sensing and processing “mini-array” chips in both largercontiguous structures and in isolated use on instrument keys, instrumentfingerboards, and instrument bodies.

With the perfection of a translucent pressure-sensor array, it furtherbecomes possible for translucent pressure-sensor arrays to be laid atopaligned visual displays such as LCDs, florescent, plasma, CRTs, etc. aswas discussed above for null/contact touch-pads. The displays can beused to label areas of the sensor array, illustrate gradients, etc. Notethat in the “tile chip” implementation, monochrome or color displayareas may indeed be built into each chip.

Returning now to the concept of a pressure-sensor array touch-pad largeenough for hand-operation: examples of hand contact that may berecognized, example methods for how these may be translated into controlparameters, and examples of how these all may be used are now described.In the below the hand is used throughout as an example, but it isunderstood that the foot or even other body regions, animal regions,objects, or physical phenomena can replace the role of the hand in theseillustrative examples.

FIG. 15 illustrates the pressure profiles for a number of example handcontacts with a pressure-sensor array. In the case 1500 of a finger'send, pressure on the touch pad pressure-sensor array can be limited tothe finger tip, resulting in a spatial pressure distribution profile1501; this shape does not change much as a function of pressure.Alternatively, the finger can contact the pad with its flat region,resulting in light pressure profiles 1502 which are smaller in size thanheavier pressure profiles 1503. In the case 1504 where the entire fingertouches the pad, a three-segment pattern (1504 a, 1504 b, 1504 c) willresult under many conditions; under light pressure a two segment pattern(1504 b or 1504 c missing) could result. In all but the lightestpressures the thumb makes a somewhat discernible shape 1505 as do thewrist 1506, cuff 1507, and palm 1508; at light pressures these patternsthin and can also break into disconnected regions. Whole hand patternssuch the fist 1511 and flat hand 1512 have more complex shapes. In thecase of the fist 1511, a degree of curl can be discerned from therelative geometry and separation of sub-regions (here depicted, as anexample, as 1511 a, 1511 b, and 1511 c). In the case of the whole flathand 1500, there can be two or more sub-regions which may be in factjoined (as within 1512 a) and/or disconnected (as an example, as 1512 aand 1512 b are); the whole hand also affords individual measurement ofseparation “angles” among the digits and thumb (1513 a, 1513 b, 1513 c,1513 d) which can easily be varied by the user.

Relatively simple pattern recognition software can be used to discernthese and other hand contact patterns which will be termed “postures.”The pattern recognition working together with simple image processingmay, further, derive a very large number of independent controlparameters which are easily manipulated by the operating user. In manycases it may be advantageous to train a system to the particulars of aspecific person's hand(s) and/or specific postures. In other situationsthe system may be designed to be fully adaptive and adjust the a personshand automatically. In practice, for the widest range of control andaccuracy, both training and ongoing adaptation may be useful. Further,the recognized postures described thus far may be combined in sequencewith specific dynamic variations among them (such as a finger flick,double-tap, etc.) and as such may be also recognized and thus treated asan additional type of recognized pattern; such sequential dynamics—amongpostures will be termed “gestures.” The admission of gestures furtherallows for the derivation of additional patterns such as the degree orrate of variation within one or more of the gesture dynamics. Finally,the recognized existence and/or derived parameters from postures andgestures may be assigned to specific outgoing control signal formats andranges. Any training information and/or control signal assignmentinformation may be stored and recalled for one or more players viastored program control.

For each recognized pattern, the amount of information that can bederived as parameters is in general very high. For the human hand orfoot, there are, typically, artifacts such shape variation due toelastic tissue deformation that permit recovery of up to all six degreesof freedom allowed in an object's orientation in 3-space.

FIG. 16 illustrates how six degrees of freedom can be recovered from thecontact of a single finger. In the drawing, the finger 1600 makescontact with the touchpad 1601 with its end segment at a point on thetouch-pad surface determined by coordinates 1611 and 1612 (these wouldbe, for example, left/right for 1611 and forward/backward for 1612).Fixing this point of contact, the finger 1600 is also capable ofrotational twisting along its length 1613 as wall as rocking back andforth 1614. The entire finger can also be pivoted with motion 1615 aboutthe contact point defined by coordinates 1611 and 1612. These are allclearly independently controlled actions, and yet it is still possiblein any configuration of these thus far five degrees of freedom, to varythe overall pressure 1616 applied to the contact point. Simple practice,if it is even needed, allows the latter overall pressure 1616 to beindependently fixed or varied by the human operator as other parametersare adjusted.

In general other and more complex hand contacts, such as use of twofingers, the whole hand, etc. forfeit some of these example degrees offreedom but often introduce others. For example, in the quiteconstrained case of a whole hand posture, the fingers and thumb canexert pressure independently (5 parameters), the finger and thumbseparation angles can be varied (4 parameters), the finger ends 1504 acan exert pressure independently from the middle 1504 b and inner 1504 csegments (4 parameters), the palm can independently vary its appliedpressure (1 parameter) while independently tilting/rocking in twodirections (3 parameters) and the thumb can curl (1 parameter), yielding17 instantaneously and simultaneously measurable parameters which areindependently adjustable per hand. Complex contact postures may also beviewed as, or decomposed into, component sub-postures (for example here,as flat-finger contact, palm contact, and thumb contact) which wouldthem derive parameters from each posture independently. For such complexcontact postures, recognition as a larger compound posture which maythen be decomposed allows for the opportunity to decouple and/orrenormalize the parameter extraction in recognition of the specialaffairs associated with and constraints imposed by specific complexcontact postures.

It is noted that the derived parameters may be pre-processed forspecific uses. One example of this would be the quantization of aparameter into two or more discrete steps; these could for example besequentially interpreted as sequential notes of a scale or melody.Another example would be that of warping a parameter range as measuredto one with a more musically expressive layout.

Next examples of the rich metaphorical aspects of interacting with thepressure-sensor array touch-pad are illustrated. In many cases there maybe one or more natural geometric metaphor(s) applicable, such asassociating left-right position, left-right twisting, or left-rightrotation with stereo panning, or in associating overall pressure withvolume or spectral complexity. In more abstract cases, there may bepairs of parameters that go together—here, for example with a fingerend, it may be natural to associate one parameter pair with (left/rightand forward/backward) contact position and another parameter pair with(left/right and forward/backward) twisting/rocking. In this latterexample there is available potential added structure in the metaphor byviewing the twisting/rocking plane as being superimposed over theposition plane. The superposition aspect of the metaphor can be viewedas an index, or as an input-plane/output-plane distinction for atwo-input/two-output transformation, or as two separated processes whichmay be caused to converge or morph according to additional overallpressure, or in conjunction with a dihedral angle of intersectionbetween two independent processes, etc.

Next, examples of the rich syntactical aspects of interacting with thepressure-sensor array touch-pad are illustrated. Some instruments haveparticular hand postures naturally associated with their playing,particularly hand drums and especially Persian and Indian hand drums(such as the tabla/baya bols, dumbek, etc.). It is natural then torecognize these classical hand-contact postures and derive controlparameters that match and/or transcend how a classical player would usethese hand positions to evoke and control sound from the instrument.Further, some postures could be recognized either in isolation or ingestural-context as being ones associated with (or assigned to)percussion effects while remaining postures may be associated withaccompanying melodies or sound textures.

As an additional syntactic aspect, specific hand postures and/orgestures may mapped to specific selected assignments of control signalsin ways affiliated with specific purposes. For example, finger ends maybe used for one collection of sound synthesis parameters, thumb for asecond potentially partially overlapping collection of sound synthesisparameters, flat fingers for a third partially-overlapping collection,wrist for a fourth, and cusp for a fifth, and fist for a sixth. In thiscase it may be natural to move the hand through certain connectedsequences of motions; for example: little finger end, still in contact,dropping to flat-finger contact, then dropping to either palm directlyor first to cusp and then to palm, then moving to wrist, all neverbreaking contact with the touch-pad. Such permissible sequences ofpostures that can be executed sequentially without breaking contact withthe touch-pad will be termed “continuous grammars.” Under thesecircumstances it is useful to set up parameter assignments, andpotentially associated context-sensitive parameter renormalizations,that work in the context of selected (or all available) continuousgrammars. For example, as the hand contact evolves as being recognizedas one posture and then another, parameters may be smoothly handed-overin interpretation from one posture to another without abrupt changes,while abandoned parameters either hold their last value to return to adefault value (instantly or via a controlled envelope).

Now a number of example applications of the pressure-sensor arraytouch-pad are provided. A natural start for a first example is that ofthe Indian tabla and baya; here the traditional bols are recognized andused to control synthesized or sample-playback sound generation. Theproduced sound can be authentic or transcend the classical instrument.Additional posture and gesture recognition can be added in either soundgeneration style to expand the available sounds and/or controladditional signal processing such as location modulation, muffling orpeaking filtering, reverb, sustain, instrument pitch, etc. Consideringhand drums more generally it is noted that whole-hand slaps are commonlyused in the technique but that the spread of the fingers in the handslap or hand after-touch of the drum head typically provide no usablecontrol. With the system described above, details of at least fourparameters of finger spread and even more on whole-hand posture inwhole-hand slaps and ongoing after-touch pressing may be used forextensive timbre variation.

Next, examples are given as to how derived parameters may be used tocontrol musical processes and lighting control, effectively allowing oneto “fingerpaint” with sound and/or light. There are a large number ofways in which six parameters of synthesizer “voices” may be controlledwith one finger. One possible example of a mapping is to use all sixparameters to control prominent features of a single synthesizer voice:

-   -   left/right position: pitch    -   in/out position: volume    -   left/right twist: waveform morphing dimension 1 (“duty cycle,”        even-harmonic content, etc.)    -   in/out rock: waveform morphing dimension 2 (“waveform        curvature,” odd-harmonic content, etc.)    -   rotation: stereo pan    -   overall pressure: filter opening

Another example is that of controlling two voices with one finger:

-   -   left/right position: pitch of voice 1    -   in/out position: pitch of voice 2    -   left/right twist: pan or filter opening of voice 1    -   in/out rock: pan or filter opening of voice 2    -   rotation: relative volume balance of voice pair    -   overall pressure: total volume of voice pair

By assigning pitch to an aspect of physical contact that isgeometrically large (i.e, position on the pad), it is possible to get agreat deal of accuracy in pitch control. In potentially typically caseswhere pitch choices are to be associated with traditional scales, thepitch control parameter may be quantized into discrete steps and eachstep assigned to a note in a scale or melody. At the point of contactwithin a selected quantization interval, a small “vibrato” neighborhoodmay then be defined so that wiggling the finger position is mapped to avibrato-range pitch variation (as on a violin string).

If the spatially-quantized positions are mapped to notes in a melody, itis possible to set up mappings for several musical phrases or in fact anentire melodic line start-to-finish. In the latter circumstance, it maybe desirable to either “page” the pitch assignments to give up one ofthe position parameters for sound control or instead use it for layingout the melody geometrically as per a sheet of music; here the spatialquantization may be uniformly spaced or under limited degrees beproportional to the pitch duration of the associated note. Thesheet-music layout is particularly interesting because it allows theperformer to concentrate extreme dexterity in the timbre and timingexpression of a melody without having to devote very much effort orattention to the selection of pitch value. The resulting allocationshift of performer attention is very valuable as the amount ofexpression and variations in timbre are often what distinguish aspellbinding performance from a run-of-the-mill performance.

Although purist musicians may scoff at the release from pitch selectionstruggles endemic in musical instruments over the centuries, they arealso known to spend thousand of dollars on finest-instruments that allowadditional nuances of expression and spend many, many years of theirlives making pitch selection efforts nearly as subconscious as thisinstrument approach does. This class of instrument controller, then,allows those years of skill development to be devoted directly toperfecting advanced degrees of musical expression, potentially higherthan may be achieved with conventional human life spans, traditionalreal-time instruments, and orchestra-conductor protocol.

Leaving higher callings in music for the moment, it is also possible touse the pressure-sensor array touch-pad for lighting control,particularly multi-channel lighting and/or motor-controlled (any one ormore of pan, tilt, zoom, gel, pattern-gel orientation, etc.) lighting.In multiple-light control situations, regions of the pad may bequantized into cells, each associated with a particular light andparameters within the region, controlling any of: light, brightness,position, zoom, gel, gel-pattern-orientation, etc. What can beespecially interesting in performance is to combine music processcontrol with lighting control. Some postures, gestures, or pad-regionsmay be exclusively devoted to only music control or only lightingcontrol parameters, but other postures, gestures, or pad-regions may beset up to intermingle and/share parameter assignments between music andlights.

It is also known to be possible and valuable to use the aforementionedpressure-sensor array touch-pad, implicitly containing its associateddata acquisition, processing, and assignment elements, for many, manynon-musical applications such as general machine control and computerworkstation control. One example of machine control is in robotics: herea finger might be used to control a hazardous material robot hand asfollows:

-   -   left/right position: left/right hand position    -   in/out position: in/out hand position    -   in/out rock: up/down hand position    -   rotation: hand grip approach angle    -   overall pressure: grip strength    -   left/right twist: gesture to lock or release current grip from        pressure control

A computer workstation example may involve a graphical Computer-AidedDesign application currently requiring intensive mouse manipulation ofparameters one or two at a time:

-   -   left/right position: left/right position of a selected symbol in        a 2-D CAD drawing    -   in/out position: up/down position of a selected symbol in 2-D        CAD drawing    -   left/right twist: symbol selection—left/right motion through 2-D        pallet    -   in/out rock: symbol selection—up/down motion through 2-D pallet    -   rotation: rotation of selected symbol in the drawing    -   overall pressure: sizing by steps    -   tap of additional finger: lock selection into drawing or unlock        for changes    -   tap of thumb: undo    -   palm: toggle between add new object and select existing object

Clearly a symbol can be richly interactively selected and installed oredited in moments as opposed to tens to hundreds of seconds as isrequired by mouse manipulation of parameters one or two at a time andthe necessary mode-changes needed to change the mouse actioninterpretation.

2.1.6 Multi-Parameter Instrument Keys

The famous multiple tape-loop Melletron product had keys which served toa rough extent as per-note volume controls, allowing valuable relativevoice level variations. Robert Moog patented a key with atwo-dimensional touch sensor on a keyboard key surface. The presentinvention allows for the synergistic combination of these technologiesso as to create a three-parameter controlling key particularly suited tovowel-choir synthesis and other applications, next extends this toinclude more arbitrary instrument keys (such as those on a woodwind),and finally develops multi-parameter sensing keys further byincorporation of the aforementioned pressure-sensor array touch-pad oneach key.

When voice choirs are used as instrumentation rather than the delivererof libretto, the principal parameters are typically the vowel sound usedand the relative amplitude of each vocal line. If these parameters wereto be controlled by a keyboard, and for the moment if unisons of two ormore vocal lines were excluded (unisons will in fact be handled later),each vocal line would be at a different pitch from the others. Thisallows at any particular instant specific keys on a keyboard to beuniquely associated with one vocal line apiece. As with the now somewhattraditional Melletron, the displacement of key sounding the note of aparticular vocal line then may be used to control the volume of thatvocal line. By incorporating a two-dimensional touch-pad controller toeach key, it is also possible to select and in fact vary the vowelsound. In phonetics and vocal pedagogy it is well known [Appelman,Winckel] that the quality of the vowel is largely determined by thefrequencies of resonances produce by the vocal cavity. In fact, the fullrange of realistic vowel sounds may be created by passing simplesawtooth or narrow-width pulse oscillator waveforms into a pair of bandemphasis filters, the vowel sounds varying as the filter emphasisfrequencies are varied. FIG. 17, adapted from Winckel, illustrates theregions of vowel sounds associated with particular resonant frequencycombinations in vowel sound production. Clearly there are twodimensions, then, which control vowel quality at this level ofapproximation, and further the surface of the key may be viewed as ametaphor for the plot of FIG. 17. Further details of effective choirsynthesis and variations upon it are discussed later, but thesynergistic value of the two-dimensional touch-pad key surface and keydisplacement as sources of control signals for choir synthesis isclearly established. In fact, this three-parameter per individualsynthesizer “voice” may be very valuable in at least two additionalsituations.

In a first of these additional situations, it is first noted that intraditional multiple-instrument orchestration, the principal parametersare volume and timbre. Using the aforementioned three-parameter keyarrangement, key displacement may again be used for per-note volumecontrol, leaving the remaining two dimensions for timbre control. Wesseland others have shown empirically that continuous multidimensional“timbre spaces” are useful organizations for analyzing and executingorchestration aspects of timbre assignment. Often two-dimensional timbrespaces offer a more than rich enough environment to be very useful. FIG.18, adapted from Wessel, illustrates some example two-dimensional timbrespaces from traditional instrument orchestration. Again, a metaphor maybe made between these two-dimensional graphs and the two-dimensionaltouch-pad key surface. The implementation on the synthesis side may beimplemented by methods as simple as volume cross-fading of sampledtraditional instruments (and/or synthesized sounds) to methods assophisticated as morphable numeric instrument models.

The second additional situation also pertains to so-called model-basedsynthesis (as employed in the Yahama VL1) but over a lesser range oftimbre variation, in fact a range typically within the scope associatedwith a single instrument rather than a multi-instrument orchestrationenvironment. Model-based synthesis typically has an abundance ofparameters and a dearth of effective methods for controlling them.Selected parameters, in fact, are controlled with global controlinterfaces such as a wheel, joystick, or breath controller. Because ofthe need for associating parameter control with each note, rather than agroup of notes, the best model-based synthesis engines then have beenmonophonic (i.e., only producing one note at a time). The invention'sprovision of a keyboard with the availability of three parameters tiedspecifically and independently to each key is an ideal solution to apolyphonic model-based synthesis instrument.

It is noted that choir synthesis, dynamic timbre-space basedorchestration, and polyphonic model-based synthesis instruments requirethe synergistic combination of key displacement and key surfacetouch-pad, while the said combination also is fully capable ofimplementing Moog's original vision for two-dimensional synthesiscontrol (filter parameters, oscillator waveforms, etc.) and as analternative implementation to MIDI keyboard channel pressure whichtypically requires each active key to be fully displaced.

The invention also provides for the application control discussed aboveto be enhanced yet further by placing a pressure-sensor array touch-padon each key. In the limit, this would allow each key to derive up to sixparameters for each point of contact on a key and even multiple pointsof contact (i.e., more than one finger) per key. Although custompressure-sensor array touch-pads could be crafted for the keys, it isadvantageous to employ the aforementioned pressure sensing andprocessing “mini-array” chips. In fact, applications to key surfacescould be used to dictate the canonical dimensions of the chips, forexample the width of the top surface of a black key and a length that isa least common multiple of a black key surface length and a white keysurface length.

FIG. 19 shows an example of keys from a traditional Western keyboardfitted with multiple uniformly-sized pressure-sensing and processing“mini-array” chips. These chips may be interconnected using thenetworking features described earlier. Alternatively, specialpressure-sensing and processing “mini-array” keys may be made withoutthe chip as a tiling sub-component; these would be networked in the samefashion. As to how the uniformly-sized pressure-sensing and processing“mini-array” chips could be applied to the keys, FIG. 19 first shows thecollection of key shapes 1900 used to make a conventional Westernkeyboard. There are five types of white key shapes (1901-1905) and oneblack key shape 1900 used, although the black key 1900 may have a taperleading from its widest base area 1906 a to a narrower top area 1906 b.Each of these white keys may be viewed in general terms as thecombination of two adjoined areas: one forward area rectangle ofdimensions 0.75″ by 1.75″ and a rear area-bar typically at least 0.375″wide (varying with different styles of keys). One example black key hasa taper leading to a top surface 1906 b of 0.25″ wide by 2″ deep, andanother example black key has a lesser taper with a top surface 1906 bhat is 0.3125″ wide by 2″ deep. As one example, a sensor size of 0.25″wide by 0.58″ deep could be tiled on the six keys 1901-1906 according tothe arrangement of 1910. As another example, a sensor size of 0.25″ wideby 0.75″ deep could be tiled on the six keys 1901-1906 according to thearrangement of 1920. As a third example, a sensor size of 0.75″ deepcould be tiled on the six keys 1901-1906 according to the arrangement of1930.

A point not discussed yet—though relevant to all the controllers—is onethat is especially relevant to all forms of multi-parametertouch-sensing keys: that is the perceptual trade-off between noteduration and the perception of timbre detail. In short duration notesthe ear is not able to gather much information about the timbre of thenote, while in long notes the ear typically examines the timbre, as wellas any inherent harmonic animation therein, in considerable detail andbecomes easily turned away when there is no variation, or easily learnedpredictable variation, in harmonic content over time. Themulti-parameter touch-sensing keyboard is thus well-targeted for thisphenomenon in hearing. On rapid notes multi-parameter touch-sensing keysmay actually be played with increasing degrees of timbre-controlarbitrariness, while longer notes may be played with a great deal oftimbre and amplitude variation. Although two degrees of freedom affordby the Moog key is helpful in adding per-voice expression forlong-duration notes, the three degrees of freedom provided by theaforementioned techniques in practice seems to be a minimalcontrol-dimensionality threshold for useful musical expression. Aventure as to why two parameters are not enough could start with thefact that there is great importance in relative volume variation betweenvoices—this leaves only one parameter then for timbre variation whichquickly bores the ear; adding another dimension allows for moresophisticated temporal interplays and variations over time in timbrequalities. Empirical support for this is seen in the fact thatdiscussions of “timbre space” and “sound color” in the literature devotea minimum of two-dimensions to timbre. A venture as to why the interplayof two timbre dimensions itself is a minimal control-dimensionalitythreshold for timbre could resort to an abstraction of FIG. 17: humanhearing is attuned to speech which is in turn a sequence ofphonemes—each phoneme, roughly, is a vowel sound modulated in timeaccording to some consonant aspects and supplemented by, loosely,“percussive” effects in other consonant aspects. With respect to timbrein phonemes, the ability of human hearing to follow and distinguishvowels and their modulation (including diphthongs) is largely centeredon essentially independent variation in the two formats. Thus, it couldbe postulated that amplitude variation together with two-dimensionaltimbre variation engages the speech center of the brain in a full andnatural way. As with speech, words and phonemes spoken quickly cannot bediscerned with nearly as much expression as words and phonemes spoken inlong duration can, and in long duration phonemes the ear is pleased withexpressive fluctuations in timbre and amplitude.

Finally, as to the handling of unisons (and the related problem ofmelodic line pitch crossings of uncommon timbre), in usual practice (andprior to the invention) these are typically addressed by use of multiplekeyboards or by a split of keyboard ranges into independentlyinterpreted zones. The addition of proximate keyboards and superimposedkeyboards as afforded by the invention significantly enhances thepractical extent to which and ease by which unisons and melodic linepitch crossings may be handled. As a simple example, if all melodiclines have timbre ranges that lie in a common range, and unisons sharingthe same timbre unisons may be naturally handled by superimposedkeyboard aspect of the invention—push the key deeper, or harder, for two(or sequentially, three, four, etc.) notes in unison all following thesame timbre control. More generally, proximate keyboards may be used topartition the notes that may be played with one hand between two, and insome cases three, distinct keyboards; this freely allows the player, inall but some pathological cases, to independently control unisons andmelodic line pitch crossings without constraint as to relative timbredifferences.

2.1.7 Video Cameras and Other Optically-Controlled Sensors

Video cameras and other optically-controlled sensors may also be used ascontrol elements within an instrument 100. As with other instrumentelements, video cameras and other optically-controlled sensors may beused stand-alone, in arrays, or as component/addendum to otherinstruments. Video cameras are especially interesting as controllersbecause of available image processing, image recognition, and imagemotion tracking utilities which have been developed for manufacturinginspection, medicine, and motion-video compression together with theability to actually display a real-time image in recording orperformance.

2.1.7.1 Non-Video Optically-Controlled Sensors

So as to devote most of the discussion to video, the case of simplenon-video optically-controlled sensors is first considered. A simpleexample is a set of photo-detectors which are used to discretely triggerone or more note, lighting, or special effect events. For example, alight-harp without strings may trigger notes, potentially together withselected stage lights and artificial fog blasts, as the fingersinterrupt light beams directed towards the photo-detectors. Anotherexample is that of a stage area with an array of light beams directedtowards an associated first group of photo-detectors: the beams toindividual photo-detectors of this first group may be interrupted, orredirected by means of reflective surfaces to a second group ofphoto-detectors, by dancers, actors, or musicians in choreographedmovement; the various deactivations and activations of photo-detectors,respectively, may trigger one or more of: note, lighting events, orspecial effects. It is noted that a later described aspect of theinvention provides for the generation of an event base on the detectionof predefined sequences of events; here then certain note phrases orpaths through the stage installation would trigger additional eventssuch as fog blasts illuminated by selected colors of light which aredistinguished by the pattern detected.

A more sophisticated use of simple non-video optically-controlledsensors is to continuously control one or more of sound, lighting, orspecial effect parameters; here the photo-detection is not one of on/offon a relatively narrow beam but rather continuous intensity variation ofa relatively wider light beam. The light intensity directed at aphoto-detector may be varied by means of varying the percentage of lightinterruption by the parts of the human body, clothing, artificial fogclouds affected by a performer, or other translucent, light-reflectiveor light-refractive objects manipulated by a performer.

In the above, the source light may exist in an environment ofperformance stage lighting or other illumination. To limit interferenceon the instrument, light sources may be any one or more offrequency-modulated, selected-wavelength operation, or minimum-intensityoperation (via inexpensive low-power lasers) methods. Alternatively, orin addition, a photo-detector may be provided with anoptically-directional shroud to limit interfering ambient light.

It is also possible to actually use stage lights as light sources forphoto-detection as an aspect of the invention. For example, a spotlightbeam may be directed, via light-reflective or light-refractive elementsoperated by performers, on to one or more photo-detectors operating ineither discrete-trigger or continuous-variation modes.

Finally, it is possible for the photo detectors to be color sensitive.This may be done any number of ways, ranging from putting color filtersover photo-detectors to using color electronic cameras and simple imageprocessing to derive average measured color. Should a camera be used forcolor or other photo-detection roles, photo-detector sites may actuallybe fiber optic paths that lead to a centralized camera element. Lightcolor directed to the photo-detectors may be varied by performers bymeans of filters, prisms, or other manipulable translucent, reflective,of refractive objects.

2.1.7.2 Video Cameras

Video cameras may be attached to an instrument for showing close-up ofthe performer's playing. The video close-up feed may be displayed onmonitors during a performance or recorded, and as discussed later,potentially involving other video sources and potentially with orwithout special effects. For movable instruments, such as guitars,woodwinds, etc. this can create an interesting visual effect as theinstrument profile will be firmly fixed in the video image while theambient visual background will move as the performer moves theinstrument. These visual effects seem to work best with instruments thathave sufficient physical inertia and/or which are supported by straps;instruments subject to significant undamped motion, such as flutes, mayactually have so much background motion that the image is uncomfortableto watch.

Video cameras, be they attached to an instrument or not, may also beused as instrument elements by processing the video image signals todegrees that range from simple average image brightness calculationthrough pattern recognition to image interpretation. In a simpleexample, the luminance signal for each video frame or interlace-field(i.e., only the odd or only even lines) may be sent to an integraterelement followed by a sample-hold element; the integrator may be furtherenhanced to not integrate during retrace intervals. The result gives theaverage brightness of the processed image. Adding two such additionalintegrate/hold elements and feeding the three the red/green/bluedecomposition of a color video signal makes an image-averaged colordetector. In these ways the same camera that produces performance and/orrecording video images may be used as a non-video optical sensor in themanners described earlier. This primitive capability, then, may allow aperformer to tilt or rotate the instrument 100 position so as to includestage lights or background images of particular brightness and/orcolors, direct or impede incoming light with the hand or objects, coverthe lens, etc., and in so doing trigger and/or continuously controlsound, lighting, or special effect events. The latter may occur when thevideo image is being displayed and/or recorded or with the video signalused solely in an instrument mode.

Far more valuable is the use of the spatial capture aspects of a videocamera. A simple example of this would be to split the image into“sub-image cells” (i.e., half, quarters, etc. of the entire video image)using various means and again deriving average luminance and/or colorinformation from each of the cells. For small numbers of cells this maybe done with analog electronics: sync detectors trigger one-shots thatgate specific integrate/hold circuits for specific intervals ofhorizontal scan lines in specific vertical regions of the image. Digitalmethods may also be used, for example: reading the image into a framebuffer which is then analyzed in the retrace interval for the nextframe, doing running calculations on the video signal as the fields arescanned, etc. Digital methods will typically scale to higher resolutionsand more complex functionalities and thus in many cases may bepreferred. Digital methods may be implemented with special dedicatedhardware or standard personal computers fitted with standard videocapture and MIDI interface cards, etc. Such personal computerimplementations may implement a number of image processing, parameterderivation, and control signal assignments in a flow virtually identicalto that of FIG. 13. These functions may be done in software run on thepersonal computer or in part or in whole by dedicated hardware boards orperipherals (for functions such as video acquisition, patternrecognition, etc.)

With the ability to process images at higher resolutions and in morecomplex ways, it becomes possible to use video in increasingly valuableways as an instrument element. By correlating higher resolution imagearea measurements, it becomes possible to recognize patterns and shapesand derive parameters from them in real-time. In fact, the same imageprocessing software structures used in pressure-sensor array touch-pads,or even exact portions of software itself, may also be used to processvideo images in real-time, replacing pressure pixel information with,for example, luminance pixel information. These algorithms may beenhanced further by exploiting available color information as well. Theshapes recognized and some of the parameters derived from them arelikely to have a somewhat different quality: the 3D-projected-to-2Dnature of camera images, gradients of luminance created by shadows andreflections, as well as the types and (potentially) ranges of shapes tobe recognized typically differ significantly from those discussed in thepressure-sensor array touch-pad context. Nevertheless, similar softwarestructures may be used to great value. Specific types of shapes andpatterns—such as written characters, particular gradients in brightnessor color, separation distances between bars and/or bar widths—may beparticularly useful variations from those shapes and patterns discussedin the context of pressure-sensor array touch-pads.

Next to be discussed are examples of how video cameras supplemented withthese capabilities may be used to trigger events and/or continuouslycontrol sound, light, and special effects.

A first example is that of recognizing the human hand posture, position,and proximity to the camera in 3-space. Simple hand orientation andposture geometry may be used to create specific control signals. In amore advanced implementation, dynamic gestures may be recognized. Thesetwo capabilities give the system, with sufficient software, the abilityto recognize a few if not many verbal hand signals; with yet moreenhancements, potentially including the ability to recognize the rolesof two hands with respect to the human body, the recognitioncapabilities could include, for example, formal ASL as well asparticular dance postures. The ability to recognize postures of hand,hand/arm, hand/arm/body, etc. allows hands, dance, “conducting” (notnecessarily restricted to formal conducting gestures), etc. to be useddirectly for the control of sound, lighting, and special effects.

In another class of examples, video cameras may recognize, and deriveparameters from, characters and/or patterns available on a stage. Suchcharacters and/or patterns may be brought before the camera, exposed andobfuscated from the camera; the camera may be turned towards thecharacters and/or patterns, etc., resulting in derived parameters andissued control signals. Stage cameras may also be used to recognize andtrack the location and some aspects of body orientation and posture ofperformers, deriving parameters and issuing control signals from theseas well.

In each of the above examples, it is noted that the use of two or morecameras, either in stereoscopic layout similar to those of human eyes orin an orthogonal layout (i.e., forward facing camera and overhead cameracovering the same 3-space region), may be used to resolve 3D-to-2Dprojection singularities in the pattern and shape recognition andprocessing.

As a third class of example, recent developments have allowed for therecognition of human facial expressions from video images and evendegrees of lip reading. These recognition and parameter derivationmethods may also be adapted in the invention to provide the ability forthe human face to be used as a controller for sound, lighting, andspecial effects. Simplified systems can be created to recognized andparameterize a few selected expressions or to recognize and measuregeometric variations in specific areas of the face.

From a formal, traditional music perspective, much of the above mayappear to be gimmickry with meaningful application at best in avantgarde installations or modern play products. In one response to this,directed on hand posture capture, it is noted that the hand in 3-spaceis clearly the most physically expressive aspect of the human body andis used to control almost all musical instruments but by very restrictedgeometric means. Freeing the hand to move unrestricted allowsconsiderably more expression to be captured. Further then, as a fourthexample, advances in cost reductions for video cameras and signalprocessing can make it possible for an array of cameras to be devoted toa traditional instrument controller, such as a keyboard, drum head, orflute key array (as well as, for example, a pressure-sensor arraytouch-pad) so as to capture hand expressions that cannot otherwise becost-effectively captured from the instrument controller.

Final, a brief preliminary discussion is provided here on thesignificant role of video in compositional and performance semiotics.For many years music, dance, art, film, plays, literature, poetry,linguistics, and other fields have come under study and compositionalmethods involving common abstractions or “signs” that lie within andamong their works and idioms. More will be said later about theinvention as a whole as an environment for more significantly exploitingsemiotics as a compositional and performance tool. However, videocameras used as an instrument element, either with or without the videostream being displayed or recorded, offer a special role in the creationof semiotic elements because they may be used to link visual symbols ofobject and body to sound, lighting, and special effects which in turnmay have assigned and/or intrinsic semiotic content.

2.1.8 Singing and Speech Detection, Recognition, and Parameterization

Speech recognition systems have become increasing accurate andinexpensive. These technologies can, in many valuable ways, be adaptedto also recognize sung words and/or phonemes. Recognized words orphonemes may be used to trigger any of sound, lighting, or specialeffect events, while existing pitch detection and amplitude followingtechnologies (as found, for example, in the early Roland CP-40 productor in the more modern MidiVox Synchro Voice product) may be used toderive continuous control signals. In addition, inter-event timers maybe used to measure individual word and/or phoneme duration.

These singing and speech recognition capabilities together with theirparameterization also have significant potential value in theaforementioned creation of semiotic elements because they can be used tolink verbal linguistic events and expression to sound, lighting, andspecial effects which in turn may have assigned and/or intrinsicsemiotic content.

2.1.9 Air Pressure, Air flow, and Air Turbulence Sensors and Transducers

Air flow, or “breath,” controllers for musical instruments are known andhave been employed in electronic woodwind-like controllers. It is aprovision of the invention to include these along with air pressure andair turbulence sensors and transducers as elements of an instrumententity 100. In particular, air pressure-sensors can be attached to airbladders to form a particular kind of pressure or squeezing controller.Air pressure-sensors can also be introduced into a wind instrumentinterior in an instrument where subsonic variations in ambient pressureoccur as the instrument is played.

Traditional wind instrument players often invoke air turbulence effects,such as transient “chiffs”, tongue trills, overblowing, etc. Airturbulence is then also a candidate control interface for use in anelectronic instrument entity 100. Air turbulence sensors may be craftedin various ways, including by means of signal processing the output ofany one or more of air flow and/or air pressure-sensors. A simpleexample would be to define a high-pass cut-off frequency for air flowand/or air pressure variations and another (higher) low-pass cut-offfrequency for the lowest musical “pitched” frequencies; the energy inthe remaining band of frequencies would be a crude measure or airturbulence. In a more sophisticated implementation, an array of airpressure-sensors can be distributed throughout a wind tube andsensor-array signal processing techniques can be used to separateturbulence signals from environmental acoustic noise, standing waves inthe tube, etc.

2.1.10 Clothing, Jewelry, Skin, and Muscle Sensors

Sensors on the human body have been used in some dance performances tocontrol sounds. The invention provides a generalization of this forsynergistic use in conjunction with others of its aspects.

Sensors may be attached to the human body by means of clothing, jewelry,straps, adhesive pads, etc. These sensors can be of a variety of types:position, motion, optical, skin resistance, muscle activity, etc. andmay be used to capture body position, posture, activity, environment,etc. and convert these into control signals used to control sound,lighting, and special effects. Sequences of control signals can also beinterpreted as gestures by recognition systems which in turn can be usedto generate yet other control signals. Interfaces to the sensors, takencollectively as an instrument entity 100, to one or more signal routing,processing and synthesis entities 120, may be done by means of radio,wireless optical, fiber optic cable, electrical cable, or combinationsor sequences of these.

Although the sensors described here taken as an instrument entity 100may be used in isolation, there is particular synergistic value in usingthese in conjunction with other instrument entities in a performance orrecording situation. For example, a particular body motion or gesture(such as raising an arm, swinging a hand, jumping, etc.) may havesignificant artistic value at a critical moment but not be captured byanother instrument entity. As another example, in recording sketchesduring a composition phase, particular body motions or gestures can beused to call attention to specific aspects of the sketch for futurereview.

2.1.11 Stage Environment and Macro-Environment Sensors

Sensors other than optical can be distributed on a stage and/or oncomponent installations on the stage (for example staircases, risers,scaffolds, sculptures, props, etc.). Sensors can also be used to measurelarger environments ranging from audience activity to outdoormeteorology. The sensors can include proximity, position, motion,weight, temperature, humidity, etc. and can be used to create controlsignals. As a result, these arrangements can be formalized into aninstrument entity 100.

Examples of such usage include human proximity and/or interaction withprops or sculptures, tracking of artificial fog cloud migration across astage, detecting the location of performers on staircases or risers,detecting audience motion activity, characterizing room-internal androom-external meteorology (such as wind speed, wind direction, rainfall,wind and/or rainfall noise, etc.) to bring it into an aspect of theperformance.

2.2 Vibrating-Element Instrument Elements and Subsystems

2.2.1 Single-channel Audio Signal Handling

The invention provides for the inclusion of traditional group (or“composite”) audio signals such as a group pickup serving all strings ona traditional electric guitar. These can be treated as a peer to any ofthe multi-channel audio signals or of special significance because ofits timbre, functionality, or traditional use. As will be illustrated inthe discussion of layered signal processing, such a signal can beprocessed so as to create the subtle or dominate backdrop against whichprocessed multi-channel signals are superimposed. In some situations,multi-channel signals on the instrument may be combined to create asingle channel audio output, as in the case where individual piezobridge pickups are only one of a plurality of multi-channel signalsources on an instrument; simple full or partial mix-downs may beprovided for use when such multi-channel sources are not featured in amulti-channel manner so as to conserve channel usage on the generalizedinterface 110. This can be particularly valuable in complex instrumentswith many arrays of vibrating elements such as those in FIGS. 4-5 andmany others to be discussed.

2.2.2 Multi-Channel Audio Signal Handling

The use of various types of musically-oriented signal processing withelectronic stringed instruments has been common in popular music almostas long as there have been electronic stringed instruments. Typically asingle pickup is used to capture audio signals from all vibratingelements on the instrument (although there may be a plurality of suchgroup pickups on a given instrument so as to obtain different selectionsof timbre).

The invention provides for the use of multi-channel electric transducerarrangements, by which each vibrating element (string, tyne, membrane,etc.) of an electronic instrument with multiple vibrating elements isprovided with an independent isolated electrical output, and dedicatedsignal processing can be applied to the signal of each vibrating elementor incomplete combinations thereof, to achieve significantly importantmusical functions—all done in a way where the same interfaces,multi-channel signal routing and processing, and internal instrumentelectronics can be reused across a variety of instruments.

Multi-channel vibrating element pickup arrangements, by which eachvibrating element (string, tyne, membrane, etc.) of an electronicinstrument with multiple vibrating elements is provided with anindependent isolated electrical output, have been commercially availablebut in largely hidden forms, most commonly used in synthesizerinterfaces for guitars. Beyond such synthesizer interfaces, and therecent Roland VG-1 product discussed later, the usage of suchmulti-channel vibrating element pickups has been limited to roles involume equalization and imaging in a stereo sound field on only a veryfew electric guitars models. Such musically-oriented signal processingis only known to have been applied to the summed mixture of allvibrating elements of the instrument, not for individual or sub-groupsof the vibrating elements of the instrument.

Conventional signal processing can be used on each vibrating elementsignal to create “generalized pedal steel guitars” (augmenting orreplacing mechanical pedal tuning changers), instantly retunable guitars(augmenting or replacing mechanical tuning changers such as the Hip-shot“Trilogy”), multi-modal Indian sitars (where drone and sympatheticstrings can be electronically retuned while playing, allowing a morerobust mix between Eastern and Western tonality in musical form),spatially animated instruments where individual vibrating element soundsare location modulated within a stereophonic or other spatial soundfield, and mixed timbre instruments where different signal processingmethods are applied to each string.

Standard pickup elements available to implement individual pickups foreach vibrating element include piezo contact elements, installed on abridge acoustically isolated from other vibrating elements, andnon-contacting coil-based electromagnetic pickup elements. Opticalpickup products have also been devised, and a coil-less Hall-effectpickup method has been taught as U.S. Pat. No. 4,182,213. Both opticaland Hall-effect methods do not involve contact with the vibratingelement. FIG. 20 shows electromagnetic, Hall-effect, piezo, and opticalpickup methods for deriving separate audio signals for each vibratingelement of a multiple vibrating element instrument entity.Electromagnetic coils and Hall effect elements require the stringmaterial to be ferromagnetic while piezo and optical methods do not.

It is noted that a pickup localized for individual vibrating elementmust by its nature have small geometry. For the pickup technologies notinvolving contact with the string (e.g., electromagnetic coils, Halleffect, and optical) multiple small pickups can be aligned along avibrating element's length; the resulting multi-channel signal may behandled with multi-channel signal processing, selected by a switch,selectively mixed/-morphed, etc. to obtain a range of tones. In oneimplementation the selection, mixing, morphing of the pickup signals,and hence the resulting output tone, may be operated by control signals.

It is noted that excessive magnetic fields from a large number ofmagnetic pickups may make a low-mass vibrating element such as a thinstring vibration go inharmonic. Although this should be a designconsideration with a number of pickups, it can also be used to producespecial effects. The invention thus provides that one or moreelectromagnetic coils, which may or may not otherwise double as pickups,be used to issue localized DC magnetic fields of varying intensity forinducing inharmonic effects on one or more selected strings, mostadvantageously under control signal control. The coils may create the DCmagnetic fields themselves or instead cause a permanent magnet to varyits distance to the vibrating element via solenoid structures.

The sloped bridges of sitars and other twanging/buzzing stringed Indianinstruments have not to date lent themselves to individual piezo bridgestructures. This is not impossible; the invention provides forindividual miniature sloped bridges, one for each string, to be embeddedwith its own piezo pickup element. Such bridges can also be used withnon-string vibrating elements, such as bars and tynes, to create newtypes of sounds. This method can also be adapted to the very gradual andsofter sloped body contact of certain African harps whose strings buzzagainst a typically animal fur-covered harp body. Alternatively, FIG. 21shows how an off-bridge buzz-plate, such as those provided by Biax tosimulate a fretless based sound with a conventional fretted bass, may becombined with a piezo bridge sensor in replacement of a gradientbuzz-bridge so as to permit the use of non ferromagnetic strings.

2.2.3 Vibrating Element Excitation

The use of “controlled (acoustic) feedback” with electronic stringedinstruments has been in common use in popular music since at least the1960's. It has been possible to replace the acoustic excitation ofstring resonance with electromagnetic excitation (as embodied by theHeet Sound E-bow) for some time, but only for one string at a time andvia hand-held mechanically operated apparatus. The practice ofelectromagnetic excitation in non-stringed musical instruments withvibrating elements is not currently known.

The invention presents a system using electromagnetic excitation of thevibrating elements of an electronic instrument to produce controlledfeedback relationships with signal processing control of the feedbackcharacteristics, typically hands-free as desired, with either standardparts (for inexpensive mass manufacture and retrofit) or morespecialized parts (to provide additional features).

FIG. 22 shows the basic idea of controlled feedback as used in recentcontemporary music circa 1960. A vibrating element 2201 within theinstrument is coupled (by electromagnetic, optical, or mechanical means)to an electrical transducer 2202 (electromagnetic, optical, Hall-effect,piezo, etc.) which converts the vibration to an electrical signal 2203.The electrical signal 2203 is applied to a power amplifier 2204 whichdrives a loudspeaker 2205 which is acoustically coupled 2206 (by meansof air, mounting apparatus, etc.) to the vibrating element 2201. Thiscreates a feedback arrangement allowing vibrations of specificfrequencies to resonate within the resulting closed-loop system. Byexciting or damping the vibrations of the vibrating element 2201,changing the characteristics of the frequency and/or phase response ofthe speaker's power amplifier 2204, and/or changing the characteristicsof the means of acoustic coupling 2206 (as in changing the distancebetween the vibrating element 2201 and speaker 2205), the “controlledfeedback” methods used in popular music are obtained. Note, however,that the required acoustic coupling characteristics are affected byfactors such as volume level (typically this must be relatively high),speaker/room geometry, room acoustics, and other difficult to controlfactors that can often be unpredictable liabilities.

The invention provides for an approach to replacing the acousticexcitation component of this process with electromagnetic excitation.FIG. 23 shows an example implementation of simple approach for replacingacoustic excitation of a vibrating element with electromagneticexcitation. In this case the vibrating element 2201 must beferromagnetic, although the transducer need not use ferromagnetic meansitself. Here, rather, the acoustic coupling 2206 is replaced byelectromagnetic coupling 2209, produced by an electromagnetic coil 2208with an internal magnet or other magnetic bias that replaces the speaker2205. Without this nominal magnetic field, the string will be excitedwith a full-wave rectification as the ferromagnetic string is drawn tothe coil regardless of the direction of current flow. In thisconfiguration, the power amplifier must now match the coil's electricaldrive requirements which can differ from the speaker's, hence thespeaker's power amplifier 2204 is replaced by the coil's power amplifier2207. A signal output 2210 for subsequent amplification and signalprocessing can be taken off at the transducer. (In particular, thisoutput arrangement 2210 differs from the Heet Sound E-bow where no suchoutput is provided or relevant for that product). The result is a systemthat provides a comparable arrangement to that of FIG. 22 without therequirements of high volume level, speaker/room geometry, roomacoustics, and other liabilities of acoustic coupling.

It is also noted that as piezo elements both convert vibrations intoalternating current signals and, reciprocally, convert alternatingcurrent signals into mechanical vibrations, a piezo group element bridgepickup can be used, in lieu of a coil, either as the audio signal pickupor as a mechanical drive exciting element. Further, the signal pickupcan also be optical or Hall effect. If both the signal and driveelements are electromagnetic (coils or Hall for signal pickup, coil fordrive) undesirable magnetic coupling, not unlike that of an electrictransformer, can occur. This effect may be minimized if said signal anddrive elements are sufficiently separated and/or shielded or otherwiselocalized (for example, with a two-coil/opposite-magnet arrangement.

FIG. 24 shows various combinations of piezo and electromagneticvibrating element pickups and exciter drivers for separatelycontrollable excitation of each vibrating element. In one arrangement,the string 2400 suspended over the bridge 2401 is electromagneticallycoupled to two electromagnetic coil pairs 2411 and 2412. Each coil pairis in a standard “humbucking” arrangement with complementary magnet poledirections 2413 a, 2413 b and complementary winding directions 2414 a,2414 b so as to significantly localize the magnetic coupling regionabout the coil. Here either coil may be used as the signal source or asthe driver. The source coil can also be replaced with an optical orHall-effect pickup. In a piezo-based arrangement, the string 2400 is incontact with a piezo pickup element 2421 on the bridge 2401, and thestring 2400 is magnetically coupled with an electromagnetic coil pair2422. Here either the coil pair or the piezo may serve as the driver andthe other serve as the signal source. In cases where the piezo elementacts as the driver, the electromagnetic signal source element may bereplaced with a Hall effect or optical technology pickup.

It is noted that the invention provides for the above discussions toapply equivalently should the signal source and driver elements serve anindividual vibrating element or a group of vibrating elements. Theinvention also provides for the case where either the signal source ordriver is a single element unit while the other is a group element unit;such configurations are easily supported by the signal routing,processing, and synthesis entity 120 (referring to FIG. 1). Inparticular the invention provides for the case where a multiplevibrating element instrument has at least one of the following:individual source and individual excitation, group source and groupexcitation, individual source and group excitation, and/or group sourceand individual excitation.

Since the driving element (coil or bridge piezo) may be mounted inpermanent relation to the vibrating element, it is possible to replaceconventional means of altering the acoustic coupling with electronicsignal processing means 2211. FIG. 25 shows adding signal processing forspectral and amplitude control of electromagnetic excitation. Forexample, fixed or adaptive equalizers can be used to alter the frequencyand phase response of the signal/vibration/transducer loop, permittingadditional control over which vibrational harmonic(s) are emphasized inthe feedback. Attenuation can be used to vary the degree of feedback.Delay can be used to alter the attack characteristics of the resonancebehavior. Dynamic compressors and expanders can be used to vary the easeand dynamics of the resonance behavior. Many interesting special effectsare possible, such as using pitch-shifters alone or in combination withdelays to transfer energy between vibrational modes. In the case of adrive coil, a pitch shifter or octave divider (such as an inexpensivetoggle flip-flop) may be used to create a drive signal that is an octavelower than the string signal and thus eliminate the need for a magnet inthe drive coil. The invention provides for any driver electronics and/orsignal processing to be any of the following: internal to theinstrument, mounted on the outside of the instrument as an add-onmodule, remotely located off the instrument (particularly in the signalrouting, processing, and synthesis entity 100), or any combinationthereof.

In most electronic instruments, a single pickup serves many if not allthe featured vibrating elements. The invention provides for theapproaches discussed thus to also be applied to such instruments usingconventional components. FIG. 26 shows an example arrangement involvingmultiple vibrating elements served with a group pickup and which arealso subjected to common electromagnetic excitation using conventionalguitar pickup components. (In the example of FIG. 26 it is understoodthat signal processing 2211 may be introduced or omitted from thefeedback loop as appropriate or desired and that a group piezo bridgemay, in some constructions, serve as a driving element in place of thepickup coil.). A plurality of vibrating elements 2201.1-2201.n share acommon group pickup transducer 2202 and a common electromagnetic coil2208. This arrangement is very simple to implement and very usefulmusically for traditional electric guitarists. A simple prototype can bemade using an electric guitar with two humbucking-pickups (such as aGibson ES-335). The rear pickup can be used as the transducer 2202 andthe front pickup as the coil 2208 (the humbucking pickups assist indecoupling the coils, decreasing a parasitic “transformer” effect).Almost any power amplifier of sufficiently high enough current orvoltage drive, (for example, even a Fender Bassman tube guitaramplifier) can be used as the coil's power amplifier 2207, directlydriving the coil (despite the impedance mismatch) from the amplifierspeaker output connector.

3 Example Electronic Controller Instruments

3.1 Touch-Pad Array

Touch pad instrument elements, such as null/contact types andpressure-sensor array types described earlier, can be used in isolationor arrays to create electronic controller instruments. The touch-pad(s)may be advantageously supplemented with panel controls such as pushbuttons, sliders, knobs as well as impact sensors forvelocity-controlled triggering of percussion or pitched note events. Inthe case of null/contact touch-pads, impact and/or pressure-sensors canbe added to the back of the pad and the pad suspended in such a way thatit can be used as an electronic drum head. If one or more of thetouch-pads is transparent (as in the case of a null/contact touch screenoverlay) one or more video, graphics, or alphanumeric displays mayplaced under a given pad or group of pads.

FIG. 27 illustrates examples of single, double, and quadruple touch-padinstruments with pads of various sizes and supplemental instrumentelements. A single touch-pad could serve as the central element of suchan instrument, potentially supplemented with panel controls such as pushbuttons, sliders, knobs as well as impact sensors. In FIG. 27, atransparent pad superimposed over a video, graphics, or one or morealphanumeric displays is assumed, and specifically shown is a case ofunderlay graphics cues being displayed for the player. Two large sensorscan be put side by side to emulate the left-hand/right-hand layout ofmany hand drum arrangements such as tabla/baya, congas, etc. This isparticularly suitable for pressure-sensor array touch-pad elements wherea larger pad-area (for example 8 to 12 inches square) could beadvantageous for detailed control. Because of the extensive capabilitiesof either type of touch-pad element provided for in the invention, thisarrangement is by no means limited to percussion applications but rathereasily serves as a far more general purpose left-hand/right-handmulti-parameter controller. In variants of this arrangement that areintended specifically for tabla/baya emulation, the relative size of thetwo pads and angle of placement with respect to the floor can bearrangement to match that expected by an experienced tabla player.Instruments involving arrays of larger numbers of touch-pads can also bevaluable. Here it may be advantageous to make the pads smaller so thatthe fingers of a single hand can touch two or more pads simultaneously.

3.2 Foot Controllers

With the extensive real-time control capabilities provided for in theinvention, foot controllers can be especially valuable. They can selectpreset configurations at various points in a control hierarchy, issuenotes or chords, control timbre, alter lighting, invoke special effects,etc. In general a commercially available floor controller typicallyincludes a plurality of momentary action foot-switches, and variousvisual status indicators such as LEDs over momentary actionfoot-switches and a master status (and programming) display. Many suchproducts also include provisions for rocker foot pedals to controlcontinuous parameters, either via external connection (as with theDigitech PMC-10 and Digital Music “Ground Control” pro ducts) orinternally (as with the ART X-15 product). With the exception of theDigitech PMC-10, the control assignment and organization capabilities ofthese controller products have historically been quite limited, and asall the products seem aimed largely at issuing MIDI program changecommands, the number of foot-switches has been small. Further, therocker foot pedals control only one parameter at a time.

The invention provides for extensive elaboration over these products bysupporting any of multi-dimensional rocker pedals, arbitrary controlsignal assignment, control signal assignment organized by selectablepages, separate alphanumeric function display for each foot control(switches and pedals), pause operations, and real-time event play-backcapabilities.

The traditional way to control volume on an electronic keyboardinstrument is by a means of a rocking floor-level foot-pedal. Morerecently such pedals have been used to generate continuous-range controlsignals such as MIDI messages, though allowing the control of only onecontinuous-range parameter at a time. Many years ago a number of“volume/tone” foot pedal products were available, though none appearavailable at this writing. These products offered a rocker capabilitydevoted to controlling instrument volume supplemented with a left-righttwist capability devoted to the control of instrument tone. Sucharrangements may be used to double the number of foot controllableparameters that can be controlled in roughly the same physical layoutarea together with the bonus of allowing a foot to control twocontinuous-range parameters at once.

FIG. 28 illustrates some enhanced foot-pedal arrangements which permitsimultaneous single-foot adjustment of a plurality of continuous rangeparameters for use with floor controllers. The use of rockingfoot-pedals to control two continuous-range parameters at once may beenhanced by using one or more side-mounted spring-levers. Side-mountedmomentary-action switches have been used on rocker foot-pedals for modecontrol (products by Ernie Ball and Soloton), but side-mountedspring-levers are particularly advantageous for continuous rangeparameters that have a specific nominal value. For example, these can beused in conjunction with pitch-shifters to modulate pitch as do the footand knee levers of a pedal steel guitar, or in complementary pairs toemulate the action of a synthesizer modulation wheel or an electricguitar vibrato “whammy bar.” In FIG. 28, one or more side-mountedspring-levers 2803, 2804 may mount on either the base 2801 or rockerplate 2802 of a foot-pedal. A spring lever may directly operate a slide,geared, pulleyed, etc., potentiometer, an optical sensor, magneticsensor, pressure-sensor, etc. to produce an electrical signal. If twoside-mounted spring-levers 2803, 2804 are positioned on opposite sidesof the pedal, two mutually-exclusive parameter adjustments can berealized (as found in pedal steel knee-lever pairs and in the action ofa synthesizer modulation wheel or an electric guitar vibrato whammybar). It is also possible to mount a springed center-return synthesizermodulation wheel 2805 at the far end of the rocker plate if thearrangement and materials used forego breakage in heavy usagesituations.

Further, it is possible to add a third control continuous-rangeadjustment capability on the rocker pedal by measuring the length-axisrotation of the foot: this could be done by various methods. As oneexample, a two-dimensional “volume/tone” foot pedal with control motionsup-down 2810 a and twisting 2810 b may be modified to permit length-axisrotation of the foot 2810 c and measure it with a potentiometer orsensor. Another method would involve putting at least twopressure-sensors 2813 on the twist plate 2812 of a non-modifiedtwo-dimensional foot pedal 2811 and deriving a control signal fromthese. A third way would be to mount a springed center-returnsynthesizer modulation wheel at the far end of the twist plate if thearrangement and materials used forego breakage in heavy usagesituations. Other methods can be used for multi-dimensional footcontrollers, such as the null/contact touch-pad and pressure-sensorarray touch-pad elements discussed earlier which can be adapted for footoperation.

The invention provides for arbitrary assignment of control signals tospecific foot-switches, foot-pedals, and other foot controllers. As anexample, one or more MIDI messages could be assigned to eachfoot-switch, foot-pedal, or other foot controller as is largely done inthe Digitech PMC-10 and with other functionality as the custom messageconstruction and hierarchical ganging provided by, for example, thePeavey PC-1600 slider/button controller). A particularly valuableadditional function would be that of issuing continuous controllermessages that oppositely complement the basic control signal value: forexample, in MIDI messages where “Continuous Controller” control valueslie in the range 0 to 127, if a continuous foot-pedal position causes afirst control signal to be issued with value of “x”, it is also possibleto enable the subsequent transmission of a second separate controlsignal to be issued essentially simultaneously with a value determinedby the algebraic relation “127-x”; such complementary signals may beused for many purposes, for example prorating an audio mix between twosources, prorating modulation indices among two synthesizer voices, etc.

Stored program memory may be used to retain these assignments. In thissituation it is advantageous to allow for multiple stored programselections to be recalled, thus allowing for multiple assignment setsfor each foot-switch, foot-pedal, etc. Each assignment set could bethought of as a “page.” Pages could be copied as a whole and edited.These capabilities would be similar to those of the Digitech PMC-10 andPeavey PC-1600 products. However, because of the number of controllerassignments and the diversity of possibilities it is desirable to addphysically adjacent to each foot controller an alpha-numeric displayindicating the current assignment and status of that controller: inparticular, for each given selected page, each controller display mayshow one or more of the currently assigned function, the currentvalue(s) transmitted or last-transmitted, any additional identifyinginformation such as short-hand names or relationships with othercontrollers, etc. LEDs may be provided for quick reference as to whichfoot-switch and which continuous foot controller (pedal, touch-pad,etc.) were last operated; as an enhancement these LEDs could be bi-colorand of the two LEDs lit at a given instant (one for last foot-switchused, the other for continuous controller used), one color (i.e., green)could be used to indicate to overall last operation while a second color(i.e., red) would be used to illuminate the remaining lit LED.

The operation of a foot-switch may be assigned, under stored programcontrol, to issue one or more simultaneous control signals, or shortburst of contiguously-sequential control signals such as a group of MIDImessages. These control signal events may occur on the depression of thefoot-switch, its release, or both. The foot switch may also beconfigured to operate in a toggle mode using a divide-by-two counter andmessages can be issued on each toggle transition. These useful featurescan be found on, for example the Digitech PMC-10, but a number of usefulenhancements are provided for by the invention. One enhancement would beto allow any specific pedal to independently operate in a generalizationof toggle mode to permit a round robin selection of 3 or more states(for example “off,” “slow,” “medium,” “fast”). Another enhancement is toallow a more complicated state transition map involving a group offoot-switches. Yet another enhancement is to permit timed events to beissued. The simplest of these would be timed pause operations betweencontrol signal events, while a more enhanced implementation would permitreal-time control event play-back capabilities to be assigned to afoot-switch. Such real-time event sequences could include not only notesequences but also trajectories of continuous parameters (for example,exponential transients or linear ramps). Further, the invention providesfor the issuance of the same selection of possible control signaloptions upon incoming or outgoing page-change events during a storedmemory page change.

Finally, larger foot controller assemblies with appropriateorganizational and ergonomic layout are advantageously provided for bythe invention. Among the factors here are overall ergonomic operation,putting some foot controlled elements closer to the user for fast orintimate use with others farther away for background or occasional use,and an overall physical and operational organizational hierarchy. Inimplementing such hierarchies, each full stored program page can involveone or more sub-pages which also be used as a part of other full storedprogram page. Although such a sub-page can in general be assigned to anyfoot operated control element, it typically would be useful to confineeach sub-page to a pre-defined reusable geometric region in the overallfoot controller layout. Further, the invention provides for sub-pages tobe changes within an active full page.

FIG. 29 shows some example layouts involving 2 geometric regions for amoderate number of foot operated controllers 2920 and 4 geometricregions for a larger number of foot operated controllers 2940. Thesmaller arrangement 2920 features a rocker pedal with two side-mountspring controllers 2903 and two rock/twist pedals 2900 as well as twogeometric regions—one proximate 2921, another remote 2922—offoot-switches 2905. Each foot-switch and pedal is provided with analphanumeric display 2906 and a last-operation indicating-LED 2907. Thelarger arrangement 2940 features an advantageous layout of two proximategeometric regions 2941, 2942 of foot-switches, two increasingly remotegeometric regions 2943, 2944 of foot-switches, one proximate rock/twistpedal 2900, remote pedals of rocker only 2901, single side spring lever2902, and double side spring lever 2903, as well as a foot or toeoperated touch-pad 2904. Each foot operated controller is also providedwith an alphanumeric display 2906 and a last-operation indicating-LED2907. The layout used in the larger unit permits logical association ofgroups of switches and pedals in a wide variety of contexts. In eitherthe smaller or larger arrangement, the more remote controllers can beput on progressively higher risers to create a staircase layout. Thesearrangements permit for an arbitrary logical hierarchy of page andsub-page recall control and arbitrary assignment of which buttons may beused to do this. In some cases it may be desirable to have an additionalsummary display showing the selected page and sub-page status in onelocation at a glance.

3.3 Multi-Tier Proximate/Superimposed Keyboards

The proximate and superimposed keyboard elements described earlier canbe combined to create a powerful enhanced keyboard controller. In anexample implementation, an arrangement of three proximate keyboards suchas shown in FIGS. 3A-C may be brought together in a common unit. Thisunit may also advantageously include one or more of any of sliders,knobs buttons, joysticks, touch-pads, strum-pads, impact sensors, etc.Further, it is noted that any of the keyboards here may be either of astandard variety or any of the more advanced keyboards described later(miniature, superimposed, multi-parameter keys, pressure-sensor array,etc.). It is also noted that this technique may be applied to othertypes of keyboards with applicable types of key geometry.

3.4 One-Hand Enhanced-Drum-Roll Controllers

The invention provides for one-handed methods of performing drum-rollswith some advanced capabilities. The basis of the method involves theproximate location of two electronic impact sensors and/or touch padsoriented to be facing each other, but the method can also be used withacoustic drums. The arrangement can be small in scale, i.e., played witha single finger, or larger to be played with hands, beaters, mallets, orsticks. FIG. 30 shows an example large-scale arrangement of two impactsensors and/or touch pads for executing one-handed drum-rolls andderiving large amounts of control information. The figure illustrates alarger-scale arrangement of two impact sensors and/or touch pads 3000 a,3000 b supported in the method's configuration by, for example,supporting beams 3003, 3004 connecting to a common suspending clamp 3005on an instrument-stand beam 3006 on one side and joints 3007, 3008 tothe sensors and/or pads 3000 a, 3000 b on the other side; though,clearly, other mounting arrangements are possible. The sensors and/orpads 3000 a, 3000 b are separated from one another by a distance thatpermits a beater 3010, mallet, or stick to be held in one hand at thefar end 3011 and rapidly rocked back and forth between the two sensorsand/or pads so that the beater head 3012 impacts the sensors and/orpads. The beater may also be held at its center of mass or geometry 3013and vibrated so that both the beater head 3012 and end tip 3014 of thefar end impacts the sensors and/or pads; in this technique the playermay orient the beater motion so as to simultaneously impact one impactsensor and/or pad with the beater head 3012 and impact the other impactsensor and/or pad with the beater end tip. In this playing technique itis advantageous to have provided for some regional differentiation ofthe impact sensors and/or pads; null/contact pads, for example can dothis. Another arrangement is that of two impact sensors, one for thecenter area 3001 of an impact pad 3000 and the other for the outer rimarea 3002. With the ability to differentiate regions of impact, and evennon-impact applied contact regions and pressure, the portion contactedby the end tip and head can be differentiated. Further enhancement canbe obtained by using a beater endowed with sensors; these can providecontact localization information, as well as hand grip information,which may be used independently or in correlation with the informationgenerated by the pads 3000 a, 3000 b. The resulting arrangement allows aperformer with one hand to do a wide range of percussion and othercontrol actions, leaving the other hand free for playing anotherinstrument entity or expressing visual gestures during performance.

In a smaller scale implementation, one or more fingers can be used inplace of a beater. This arrangement can be treated as an instrumentelement in itself to be used as part of other instrument entities.

Regardless of scale, it is noted that two such arrangements can becolinearly co-located but in 90-degree rotational offset. This creates arectangular cavity for beats, fingers, etc. to be inserted and vibrated,and additional degrees of control. This can be generalized intoarbitrary polygonal cross-sections (triangles, pentagons, hexagons,etc.).

3.5 Video Hand Position and Gesture

A camera with appropriate real-time image processing may be usedsimultaneously or mutually exclusively as an instrument element as wellas a video feed source for recording or performance. As such the cameramay be treated as an instrument element mounted on an instrument entity,but can also be used as a self-contained instrument entity. For example,a camera could be aimed upwards and surrounded by illuminating lights. Aperformer can activate and control this instrument entity by putting ahand over the camera and executing various positions and gesturesrecognized by the image processing capabilities.

3.6 Video Stage Tracker

A camera may also be used to transform visual information observed froma stage into control signals. The relevant image processing andrecognition capabilities may advantageously include identifying andtracking performer location and motions.

4 Example Adapted Instruments

This section discusses example manners and methods the inventionprovides by which a number of traditional vibrating element instrumentscan be enhanced by incorporating various synergistic combinations oftraditional components and the invention's instrument elements.

4.1 Autoharp

A traditional autoharp incorporates a plurality of strings, tuned toselected notes in a chromatic scale, which are selectively damped bymechanical damping bars with cut-outs in the damping material that allowonly selected strings to sound. A player selects and activates a damperbar associated with a chord and strums a portion or all of the strings,and only the undamped strings, namely those associated with the voicingof the chord, sound. Although at times considered a lower folk orbeginning instructional instrument, the basic arrangement of theautoharp can give rise to a powerfully flexible instrument.

In its simplest provision, the invention provides for an autoharp to besupplemented with sliders, switches and buttons for issuing controlsignals. In particular, a select group of buttons or contacts can beoperated by, or in conjunction with, the mechanical damper bars. Thisgroup of buttons or contacts may be used to control at least one of thefollowing: issued note control signals for sound, lighting, and/orspecial effects, note assignments to one or more strum-pads, and/or theamplification of individual strings. The individual strings of theautoharp may have one or more of the following: a common pickup for theentire group of strings, a plurality of smaller pickups associated withsub-groups of strings, or a full plurality of individual pickups foreach string. The pickups may be any of electromagnetic, piezo, optical,etc. in their operation. In cases where a plurality of pickups isemployed, signals from groups of strings or individual strings may behandled by multi-channel signal processing as described later (forexample, treating the strings with differing degrees of equalization,chorus, reverb, pitch shift, dynamic filter sweeps, etc., and/orproviding separate noise gates, compression, limiting, amplitudecontrol, etc.). In cases where each string has its own pickup, theplucking of a particular string may further be used to trigger asynthesizer note, lighting, or special effect event, potentially usingthe amplitude of the pluck to set note velocity and potentially trackingthe on-going string amplitude and even harmonic structure variations asprovided for in the invention and described later. Strum-pads may beprovide for use in conjunction with strumming the strings or inconjunction with operating the mechanical chord dampers. Controls may beprovided for stored program recall of control signal assignments,strum-pad voicings, etc. as well as operational features such muting orsustaining of strum-pad notes, whether notes issued at the pressing of achord damper bar are released when the damper bar is released or insteadonly when a new bar is activated, etc. These control features may alsobe controlled remotely, for example, with a foot controller, and/orimplemented remotely in a separate signal routing, processing, andsynthesis entity 120.

FIG. 31 shows an example of an enhanced autoharp implementation asprovided for in the invention. A conventional autoharp 3100 with theusual arrangement of strings 3101 and damper-bar activating chordbuttons 3102 may be fitted with a long strum-pad 3103 adjacent to thetraditional strumming area, one or more shorter strum-pads 3104 a, 3104b near the chord button area, a plurality of slider controls 3105 andcontrol switches 3106, and control buttons for stored program recall3107 a, operational mode control 3107 b, or other features 3107 c.

As another part to the invention, the mechanical chord damper bararrangement may be advantageously replaced with a 12-note keyboard orsimilar arrangement for selecting which chromatic notes are allowed tosound. String damping control may be done mechanically although thisrequires damper bars to normally damp selected strings and let thosewanted strings sound only when a key or button is depressed (rather thandamping only unwanted strings when a key or button is depressed). Inthis way more arbitrary chords can be selected, chords can bedynamically changed at a resolution down to one pitch at a time, etc.Alternatively, if a separate pickup can be provided for each string,mechanical string sounding control may be replaced with electronicamplitude control. In the simplest form, all strings of various octavesof the same note are gated on and off by the depression of the key onthe keyboard associated with that note. If the key depression-depth ortotal pressure on the key is used as a volume control, the relativevolume of all octaves of each pitch can be controlled independently fromthat of other pitches. If the key further has two-dimensional touchsensing, as with a null/contact touch-pad on each key, balance betweenvarious—typically four—octaves can be continuously varied (for exampleleft/right controls the balance between octaves 1 and 2 and in/outcontrols the balance between octaves 3 and 4, thus allowing arbitrarybalance choices of the four octaves). The multi-parameter key control ofthe amplitude and mix of each sounded note is of particular value whilethe string sounds after the note is initiated. The keyboard,multi-parameter or not, can also be used to control similar aspects ofnote assignments and amplitudes of synthesizer notes initiated with eachstrum-pad.

FIG. 32 shows how the autoharp arrangement of FIG. 30 can be adjusted toreplace the chord button array 3102 and associated strum-pads 3104 a,3104 b with a keyboard 3202 and one or more strum-pads 3204 positionedover the keyboard.

4.2 Harps, Koras, Zithers, Kotos, Mbiras

The enhancements of Harps, African Koras, Zithers, Japanese Kotos,African Mbiras, and other related instruments with a large array ofhand-plucked vibrating elements are also provided for as part of theinvention. As with the above autoharps, pickups may be used for allvibrating elements, or, advantageously, sub-groups of elements, or—mostadvantageously—separately for each vibrating element. The pickups may beany of electromagnetic, piezo, optical, etc. in their operation. Theinvention also provides for the instrument to be supplemented withstrum-pads, touch-pads, sliders, switches and buttons for issuingcontrol signals and affecting internal operation and note-event handlingmodes.

In cases where a plurality of pickups are employed, signals from groupsof vibrating elements or individual vibrating elements may be handled bymulti-channel signal processing as described later (for example,treating the strings with differing degrees of equalization, chorus,reverb, pitch shift, dynamic filter sweeps, etc., and/or providingseparate noise gates, compression, limiting, amplitude control, etc.).In cases where each vibrating element has its own pickup, the pluckingof a particular vibrating element may further be used to trigger asynthesizer note, lighting, or special effect event, potentially usingthe amplitude of the pluck to set note velocity and potentially trackingthe ongoing string amplitude and even harmonic structure variations asprovided for in the invention and described later. Strum-pads may beprovided for use in conjunction with plucking the vibrating elements.

Harps, Koras, Zithers, Mbiras, and other related instruments with alarge array of hand-plucked vibrating elements often have only selectedpitches available; accidentals and extreme octaves typically are notrepresented. Many of these instruments allow for accidentals duringplaying, for example harp tuning levers and Koto string bends, whileothers, such as the Mbira, do not; in almost all cases extremal octavesare not supported at all (aside from execution of fundamental-mutingstring “harmonic chiming” to attain high octave pitches). With eachvibrating element (or, less flexibly, groups of vibrating elements)provided a separate pickup and audio channel, pitch shifting can be usedto electronically obtain pitches not provided for by the natural form ofthe instrument as well as large expressive pitch bends that may also nototherwise be possible.

FIG. 33 shows an example Koto implementation provided for in accordancewith the invention. In general a Koto includes a number of strings 3301(typically 13 to 22 in number), a sounding bridge 3302, and a movabletruss bridge 3303 for each string. The Koto 3300 shown in FIG. 33 is ofthe Vietnamese variety, traditionally strung with sixteen metal strings;in general any traditional Koto (or Chinese Sheng) can be adapted aswill be described and may have any traditional number of silk, nylon,metal, or other material strings. In this example the Koto has beenfitted with geared string tuners split into two groups 3304 a, 3304 b tofacilitate radical tuning changes and string replacement; other nominalstring tuning mechanisms, including traditional friction pegs or slipknot systems, may also be used. Each string can be given its own pickup3306.1-3306.n, at the bridge 3302; alternatively, or in addition (shouldthe string be such that electromagnetic, optical, or other non-piezopickups be applicable) at a different string location in one or morepickup housings 3305. Multi-channel signal handling as described earlierand later on can be used. The Koto can be fitted with a strum-pad 3310and may also be provided with various additional sensors and controls asdescribed earlier. Because of the unique pitch-bending arrangement ofthe Koto involving varying the tension of the string on the non-pluckedside of the movable truss bridges 3303, it may also be advantageous toprovide strain gauges on far bridge 3307 or via, for example, flexibleattaching electrical cables on the truss bridges 3303 themselves.

FIG. 34 shows an example Mbira implementation provided for in accordancewith the invention. Here a traditional Mbira 3400 with three sets oftynes 3401, 3402, 3403 secured by a bridging pressure bar 3404 is shown.Here the traditional bar bridge may be replaced with individuallyadjustable bridge elements 3406.1-3406.n which may include separatepiezo pickups for each tyne; alternatively electromagnetic or opticalpickups may be provided under each tyne. The Mbira may also be providedwith one or more strum-pads 3407, 3408 which may be full length orpartitioned into right and left segments 3407 a, 3407 b, 3408 a, 3408 bfor use with the thumbs of both hands.

Any of these instruments may also be provided with vibrating elementexcitation employing the methods presented earlier in association withFIG. 24. In addition, as the timbre of these instruments is typicallyshaped by the sympathetic vibration of unplucked strings on theinstrument, it may also be advantageous to excite, as a group orindividually, a number of vibrating elements such as tynes or stringsusing the methods described earlier; these sympathetically vibratingelectronic elements, which need not be mounted to the instrument, canthen produce audio signals that can be used for ambient effects.Alternatively, a dedicated group of sympathetically vibrating elementscan be mounted to the instrument and excited mechanically rather thanelectrically. If the sympathetic vibrating elements have individualpickups, some of the elements can be selectively turned off orattenuated so as to thin out or spectrally sculpt the ambient effect.

4.3 Single-Course Guitars and Variations

One of the most versatile instruments available for the range of timbreand expression is the electric guitar which is sadly not often usedseriously in music composition due to its origins and significant rolein popular music. (In fact, at this writing, even toy pianos are takenmore seriously than the electric guitar!) Part of the reason for theimmense range of timbre and expression is the fact that it is one of thefew instruments where both hands can be in direct contact with thestring. Another important reason is the range of timbres that can resultfrom string pickups followed by a wide degree of signal processingmethods that have been developed and can be applied. Although therecontinue to be developments in basic electric guitar themes, theinvention provides for significant enhancements of the electric guitaras a powerful instrument entity.

An important first step is the provision of separate audio signalpickups for each string; these may be electromagnetic, piezo, optical,etc. This allows for multi-channel signal processing as will bedescribed later (for example pitch shifting particular strings for bigbass notes, enhanced processing for strings playing solo lines to standout from strings playing background material, etc.). Strings may begiven one or more dedicated or shared pickups at different points alongthe string's length so as to capitalize on the different harmonicstructure and dynamics offered by different pickup locations. Aplurality of pickups dedicated to the same string or same group ofstrings can be selected or mixed, potentially in adjustable phaserelationships, statically and/or varying in time, on the instrumentand/or externally. Further, selected strings may be excited byelectromagnetic, piezo, or other methods to give a continuously soundingbowed effect whose inter-note attack can be controlled by variousfretting techniques. Additional strings arranged to serve as a harpelement, bass notes as on an arch-lute, or for sympathetic vibration mayalso be provided, as may tynes or other vibrating elements used insimilar ways. Strum-pads, sensors, sliders, joysticks, buttons,touch-pads, actuators, etc. may also be added to issue control signalsto any of signal processing, lighting, synthesizer, or special effects.Similarly, video cameras can be used to generate control signals and/orfor video image feeds in performance or recording.

FIG. 35 shows an example electric guitar implementation in accordancewith the invention based on a Gibson model ES-335 guitar or otherinstruments of that style. The invention's enhancements shown can beadded on as modules, added collectively, or built-in. The core guitar3500 features six strings 3501.1-3501.6, a bridge 3502 which may provideseparate piezo pickups for each string, an adjacent hexaphonic pickupmodule 3503 which provides separate electromagnetic, optical, or otherpickups for each string, and additional individual string pickup module3404 for the three lower pitched strings 3501.4-3501.6, two sharedpickups of a two-coil humbucking design near the neck 3505 a and nearthe bridge 3505 b, shared pickup volume and tone controls 3506, and ashared pickup selector switch 3507. Individual strings may beelectronically excited by means of any of a hexaphonic electromagneticdrive unit 3508, a hexaphonic pickup used as a drive unit (or designedas a drive unit) built into the forward shared pickup 3505 a, or bymechanical excitation via the piezo elements in the bridge 3502.Individual string excitation, or even string activity in general, can bevisually indicated by an LED (preferably high-brightness) array 3509under the strings. Group string excitation may be realized by whicheverof pickup 3505 a or 3505 b is not in use, an additional module under thestings, or by group mechanical excitation via the piezo elements in thebridge 3502. Individual string pickup gain normalizing adjustments foreach hexaphonic pickup can be made available for screwdriver adjustment3511 a, 3511 b on an add-on box or built-in panel 3512, shown in theFigure as an add-on box with generalized interface connectors 3519 onits back downward side. The instrument may also be provided with anarray of control switches 3513 and sliders 3514, individual stringpickup selector switches 3516, a video camera 3517 aimed at the playingarea but also useful for hand posture and gesture control, and an area3518 for additional controls, touch-pads, string or tyne arrays, etc.

FIG. 5 shows an example electric guitar implementation in accordancewith the invention based on a Gibson Explorer model guitar; theinvention's enhancements shown can be added on as modules, addedcollectively, or built-in. The core guitar 500 includes six strings501.1-501.6, a locking-nut vibrato bridge 502 a with string-tension“whammy bar” 502 b and fine tuners 502 c, a hexaphonic electromagneticor optical pickup module 503, a shared pickup 505 a near the neck andanother 505 b near the bridge, a shared pickup volume control 506,shared pickup selection, mixing, phase, and coil-shunt switches 507. Theinstrument may also be fitted with a hexaphonic electromagneticexcitation module 508 with string drive indicated LEDs 509. Alsoprovided for by the invention are controls for creating control signals:an array of switches 513, knobs 514 a, one or more expression wheels 514b, one or more joysticks 514 c, two side-by-side strum-pads 516 a, 516 bwhich can be operated as one long strum-pad, chord buttons 520 asdescribed for the autoharp but here used without strings and rather inconjunction with the strum-pads, a touch-pad 522, two miniaturekeyboards 521 a, 521 b, a miniature harp/sympathetic string set 524 withgroup or individual pickups partitioned in string triads with separatebridges 523 a, 523 b, 523 c and tuning heads 525, and a plurality ofimpact sensors. Internally the whammy bar may operate a discrete orcontinuous position sensor and the instrument may also include motionsensors (such as accelerometers) or position detectors (radio,ultrasonic, optical, etc.). The generalized interface connector area 519in this instance is shown built into the instrument.

4.4 Baroque and 12-String Guitars, Lutes, Tars, Setars, Saz, Oud,Mandolins, Mandolas

These instruments involve double-strings. In addition to the techniquesand additional instrument elements, each double-string pair may share anindividual pickup, or each string within in a double string pair mayhave its own pickup. At this writing the best mode for the latterappears to be piezo pickups at the bridge due to limitations inlocalizing magnetic fields for such close geometries but optical orother methods could be devised. With a separate signal for each stringwithin in a double string pair, either of the strings can be selectivelydisable, pitch-shifted, equalized, etc. along with other capabilitiessuch as adjustable balance, stereo spatial output, opposing locationmodulation trajectories, etc. Further, as a combined double-stringsignal would confuse audio-to-note information conversions, separationof the string signals for a given string pair enables control extractionsuch as conversion to MIDI note functions.

FIG. 36 shows an example of an adapted European arch-lute with a mix ofsingle strings and double string pairs. The European arch-lute 3600 withvarious ranges of extended fretted and unfretted bass strings is a nobleinstrument which with amplification, signal processing, andmulti-channel signal handling can contribute greatly to electronicmusic. FIG. 36 shows a close-up of the strings at the bridge 3620 for afourteen string version of the lute with six extended-length unfrettedbass strings: the two highest-pitched melody strings 3601, 3602 and thesix extended-length unfretted bass strings 3609-3614 are single whilethe remaining courses 3603-3608 are double-strung. Of the double-strungstrings the higher pitched ones are typically in unions while the lowerpitched ones are paired in octaves. It is understood than many otherstring configurations, varying in how many strings and which are singleor doubled, exist and can be adapted as described herein. In fact noelectric lutes are known at this time, thus it is novel simply toinclude a group piezo pickup in the bridge 3620 for all the strings soas to bring the instrument into the world of amplification and signalprocessing. A next level refinement would be to provide separate grouppickups for the extended bass strings and the rest of the strings sothat special equalization or other effects can be applied to the bassnotes in a manner differing from the other strings. A next level ofrefinement would be for the bridge 3620 to provide an individual pickupfor each individual single string course or string-pair, while a finalrefinement provides a separate piezo pickup for every individual stringon the instrument.

It is understood that various controls, strum-pads, etc. may also beadded in the manner described for previous instrument examples. It isalso understood that the methods described also apply to otherdouble-strung instruments such as 12-string guitars, Saz, Oud, Mandolin,etc. Many of these instruments may also benefit from an additional setof unfretted bass strings as incorporated in the traditional Europeanarch-lute.

4.5 Pedal Steel Guitars

The pedal steel guitar is a remarkable instrument in that the pitches ofindividual strings are changed as a group by a hand-held metal slide andrelatively within the group by mechanical bridge arrangement, usuallycalled a “changer,” which changes the tension on one or more selectedstrings in response to the action of a given foot-pedal or knee lever.The basic sound of the steel guitar is very attractive and it ispossible to tastefully play Bach chorales and hymns on the instrument.Years of incremental development have lead to specific standard pedaland knee lever configurations that are widely accepted. Variations aresometimes difficult to implement because of mechanical limitations toprovided adjustments. Because of the commitment involved in mechanicallyestablishing an alternate pedal and lever configuration, immenseexperience and/or a computer-aided design tool may be required to makevaluable accomplishments. By providing a separate pickup for eachstring, retuning can be done electronically, supplementing or replacingthe traditional mechanical mechanisms. As with other adaptations ofinstruments described thus far, each string can also be processedseparately or in groups as desired, allowing for mixes of timbres, andaudio-to-control signal extractions can be used to control synthesizers,signal processing, lighting, and special effects. Further, the nearlyfixed position of the picking hand and the freedom of some fingers inadapted playing techniques allow usage of miniature keyboards andstrum-pads in the picking area as well as use of the wrist to controlparameters. Information from the mechanical or electronics pedals andlevers and the steel bar position can be used to control the pitchesassigned to a strum-pad. The bar itself can have a control areabuilt-in, detecting applied pressure, for example.

FIG. 37 shows an example pedal steel guitar adaptation as provided forby the invention. A traditional pedal steel instrument arrangement 3700is used here. The bar 3702 may or may not be provided with internalsensors or controls. The position of the bar 3702 over the instrument3700 may be sensed by various means (changes in round-trip stringresistance, capacitive sensing, etc.) within the instrument 3700. Theinstrument may include a traditional mechanical changer 3740 withlinkages to pedals 3741 and levers 3742 or may be fixed if all pitchbends are to be done electronically. With a mechanical changer, thepedals 3741 and levers 3742 may be fitted with sensors that measure andconvert their displacement into control signals. Other control signalsmay be issued by a wrist operated expression wheel 3714 a (which inpractice works well for overall volume control), one or more pedals 3714b, some of which may be fitted with side-mounted spring-lever controls3714 c. The pedal rockers can be used for various control functions, butoffer a way of bending pitches electronically and holding them withouthaving to devote a foot or knee to that purpose. In contrast, theside-mount spring levers, as well as a spring loaded version of a rockerpedal, directly emulate the spring-return operation of a traditionalpedal steel guitar's pedals and levers. The pedals can also be used forthe introduction or variation of signal processing on one or moreselected strings and can be configured to make such effects whilesimultaneously changing pitch of selected string or strings. Theadaptation also provides for a strum-pad 3716 and miniature keyboard3721 in the picking area for use by idle fingers. Finally, afoot-controller foot-switch unit 3742 for use in selecting storedprograms, controlling signal processing, issuing notes or chords,operating drum machines, etc. is also provided.

4.6 Sitars

The Sitar is an extraordinary rich instrument that is well-suited forthe particular structural details of classical indian music. It includesa number of drone strings, only one or two of which can be fretted inany musical way, a single melody string, and an octave pair of unfrettedhigh pitch strings, called the “chikori” (Western spellings vary) usedfor a variety of purposes including quite effective rhythmic accents,all sharing a common sloping bridge that cause the aforementionedstrings to twang to a degree determined by the slope of the bridge. Aset of sympathetic strings with their own sloping bridge, which in sometechniques can be arpeggiated and/or used as a small harp to a limitedextent, is also provided. The Sitar features a selected combination ofboth brass and steel string types which have important essentialdistinctions in timbres.

Uses of the Sitar in Western music tend to fall into two categories: onewhere only the melody string, along with any sympathetic string action,is used, and another where the sitar's many drone strings force thetonality into the standard Indian tonal development system (rich andextraordinarily beautiful as it is). A Sitar-like sloping bridge hasbeen successfully put on a guitar (the Jerry Jones “Coral Sitar” heardin many Motown-era popular recorded sings), but all that remains is thetwang as the genius of the Sitar holistically has been omitted.

The invention provides for a powerfully rich adaptation of the Sitar bycombining the techniques described thus far with the signal routing,processing, and synthesis techniques to be described later and, as withthe previous examples, inherent aspects of the instrument.

FIG. 4 shows an adapted sitar as provided for in the invention. The coreinstrument is a standard Indian Sitar 400 with a standard melody string401.2, any one of a number of possible stringings of drone strings—heretwo 401.3, 401.4, are used, and the chikori pair 401.5 a, 401.5 b allsharing the common sloped bridge 402 a. Also part of the coreinstrument, but not showed explicitly in the Figure for the sake ofclarity, are the sympathetic strings, typically eleven in number, withtheir own sloping bridge 402 b and multi-length termination area 402 onthe neck under the Sitar's curved frets. Because of the curved frets andthe flat bridge 402 a, the drone strings 401.3, 401.4 will not soundaccurately in pitch for nearly all the frets; further, thecharacteristic extensive bending of the melody string, often theinterval of a major fourth or more, requires quite a bit of area on thesitar neck (namely the entire bottom half of the neck) and as a resultthe drone strings' use in melody is essentially nil. However, it ispossible to add an additional melody string 401.1 with its own lowerbridge; this string would be pushed up to get the bending effects thatthe original melody string 401.2 is pulled down for. Both melody stringsare provided with fine tuning adjustments such as the typical beads 430;these may also be used on drone strings and/or alternate tuningmechanics may be substituted throughout the instrument. Also provided,though not essential, is an additional set of strings 424 that can beused as a harp in addition to the limited-access aforementionedsympathetic strings. In the Figure these are shown well-removed from theneck at the top of the instrument, but the assembly 424 could be broughtforward and put in the position currently occupied by the keyboard 421.The keyboard 421 is also not essential but is easy to supportelectronically, employing the method associated with FIG. 8, inconjunction with the strum-pad 416 a located on the plucking area of theneck and the strum-pad 416 b located on the thumb-rest area of the neck;further, both the keyboard Harmonium and keyboard synthesizer arepopular in Indian classical, modal, and Ghozal folk music. Thestrum-pads are provided because of there particular potential use in theRag (Raag, Raga) form both in laying out the scale of the Rag and melodyexpositional fragments which are repeatedly drawn upon.

Important to the adaptation is the pickup assembly 403 which provides aseparate pickup for each melody string, each drone string, and eitherthe chikori pair or its individual strings. The separate outputs allowfor pitch shifting of individual strings; in particular, the pitchshifted retunings of the drone strings and chikori can be made whileplaying. If the pickup is electromagnetic, the brass strings cannot beused. There is the opportunity here for alternative stringing systems,particularly if pitch shifting of individual strings is used to createlarger pitch-shifts, but the character of the brass strings is beautifuland can be captured. One method is to use an optical pickup for thepickup assembly 403. Another more radical approach is to replace thesloping bridge 402 with a standard bridge arrangement fitted withindividual piezo pickups and to create the twanging using the off-bridgesitar plate discussed in association with FIG. 21.

The additional melody string can be tuned in union or in an interval tothe original melody string; because the have separate audio channelsthey can be processed differently or be located at different positionsin the stereo sound field. Further, the additional melody string,strum-pads, and addition string assembly serve to expand an importantorchestrational aspect of seasoned Sitar technique, namely a constantvariety of timbres and effects with attention constantly shifting amongthem. Finally, the electronic pitch shift retuning capabilities allowfor hitherto impossible tonality shifts within the Sitar environment,while the electronic pitch shift pitch-bend capabilities allow the dronestrings to obtain pitch bending and the melody strings to be harmonizedin a pitch-modulated manner.

It is also possible to carry simplified versions of the Sitar tonalityinto more Western instrument formats. FIG. 38 shows an exampleflat-necked instrument 3800 with a five string section emulating a sitarstring arrangement and several additional strings used for bass or otheraccompaniment. The sitar emulation section involves strings3801.1-3801.5 sharing a common bridge 3802 a providing a separate piezopickup for each of the strings. The first four of these stringsterminate their vibrations at the nut 3802 c and are tuned with necktuners 3851, while string 3801.5 acts as a one-string fretted chikoriand terminates its vibration at one of the mid-neck frets. The bestterminating fret may depend on the chosen tuning and thus terminatingholes for this string might be provided at several different frets, forexample from the 4th fret to the 12th. String 3801.5 requires adifferent tuning arrangement and in order to accommodate a selection oftermination points the tuner 3802 may be located on the body, perhapswith the tuning head in a protective cut-out area of the body. Anotherimportant item is that string 3801.4 is brass while the others aresteel, creating the Sitar sonority. The additional strings, shown inFIG. 38 as six in number 3801.6-3801.1, can serve as bass strings orharp strings and may be terminated on a rapid return bridge (such as theHipshot Trilogy product) to facilitate flexible use during performance,particularly because of the width of the fretted neck; alternativelythese strings may be unfretted as on the European arch-lute. Thesestrings are most likely best served by steel strings and each string maybe given its own pickup either at the rapidly retunable bridge assembly3802 b or with a body-mounted multi-channel pickup 3805.

Finally, in lieu of a sloped bridge or the arrangement of FIG. 8, it isalso possible to create a synthetic and/or enhance the actualsympathetic/buzz/twang aspect of these instruments with signalprocessing techniques. A precisely-set, short audio delay (for example 3msec, 6 msec, or 12 msec for various octaves of the note E in A440tuning) with high positive feedback acts as a resonator that easilydistorts when excited by an audio input at its resonant frequency. Thisresolutes in a swelling and diminishing twanging similar to that createdby the sloping bridge. This sort of effect can be found in one of theexample presets of the Boss SE-70 stereo signal processor. The effectcan be considerably enhanced by following this resonator with alow-speed sweeping flanger and a low-speed sweeping chorus, each withtheir own moderate amount of positive feedback, and made moreemotionally powerful by low-speed auto-panning location modulation. FIG.39 shows an example multiple-pitch sympathetic/buzz/twang resonatorfeeding an input signal 3900 into banks of short audio delays3901.1-3901.n with high resonances tuned to each selected pitch, eachfollowed by a dedicated low-speed sweeping flanger with moderateresonance 3902.1-3902.n, a dedicated low-speed sweeping flanger withmoderate resonance 3903.1-3903.n, and a low-speed auto-panner3904.1-3904.n. All autopanner outputs are summed by a stereo mixer 3905a, 3905 b to create a stereo output 3906 a, 3906 b. It is important thatall the sweep oscillators be slightly detuned to minimize repetitivediscernible patterns. The number of individual resonant pitchessupported would involve every note in the scale and perhaps an octave ofsome selected notes such as the tonic and fifth; this could range from 5to 16 pitches. It is understood that variations, simplifications, andelaborations of this example arrangement are possible.

4.7 Pipas

Like the Indian Sitar, the Chinese Pipa features a mix of string types,here involving steel, silk, and composites of these. The Pipa (and tosome extent its Japanese colleague, the Biwa) also has a rich ancienttradition yet contemporary appeal. Despite being far less known, it iscapable of a great range of sonic techniques, with a high number offormal playing techniques as compared to many other instruments.Included in the extensive technique suite are a number of body taps andimpacts made on the large front surface of the instrument.

As with the above example adaptations, the invention provides foradaptations of the Pipa that involve instrument elements of theinvention set to capture and complement the characteristics of this richand deep instrument. Again, piezo bridge pickups are felt to be the bestmode for capturing the subtle acoustic nuances of the different stringtypes, and a separate pickup for each string permits the usualmulti-channel signal processing possibilities and control signalextraction for controlling synthesizers, signal processing, lighting,and special effects. Body taps and impacts can be directed towardsimpact sensors, and the usual possible collection of extra strings,keyboards, strum-pads, touch-pads, sliders, switches, buttons, sensors,etc. may be added to the large open area for instrument augmentations.In particular, strum-pads and a bank of harp strings are especiallyapplicable due to the common use of pentatonic scale sweeps and repeatedshort melodic figures during development. Also especially useful forincorporation into Western sonic structures would be the addition of abank of bass strings and the use of signal processing as the Pipa tonalrange, though fascinating, arrives somewhat unfocused on undevelopedWestern ears unfamiliar with the instrument repertory.

FIG. 40 shows an example adapted Chinese Pipa as provided for by theinvention featuring various impact sensor arrays 4003 a, 4003 b, 4003 c,a keyboard 4004, slider array 4007, switch array 4008, touch-pad 4009,strum-pad 4010, and separate piezo pickups for each string 4001.1-4001.4at the bridge 4002.

FIG. 41 shows another example adapted Chinese Pipa as provided for bythe invention featuring impact sensors arrays 4003 a, 4003 c, strum-pad4010, an unfretted harp string array, involving a terminating bodytuning bridge 4110 a and body bridge 4110 b, and an unfretted bassstring array involving body bridge 4111 a and head tuning bridge 4111 bwith tuners 4113. The two string arrays may have individual string orgroup pickups located either in the body bridges 4111 a, 4111 b and/oron the body at an interior portion of the string vibration 4114, 4115.

It is understood that many other combinations of instrument elements arepossible.

4.8 Erhus, Dilruba, Esraj, Sarangi, Kamamcheh

Each of these bowed instruments has its own rich tradition and specialtonal qualities. Many of these instruments are used to accompany vocalsor even to replace a singer due to the vocal quality of the instrument.

The invention provides for adaptation of these instruments involvinginstrument elements of the invention set to capture and complement thecharacteristics of the traditional instrument and its musicaltraditions. In particular, in addition to the vocal quality of thesounds, bowing is a more conspicuous part of the sound as opposed toWestern bowed instruments which encourage burying the perception ofbowing logistics in favor of overall smoother tones.

Again separate pickups may be used for each string: electromagnetic,piezo, and/or optical as appropriate for the type of string material,mounting arrangements, and other engineering considerations. A separatepickup for each string permits the usual multi-channel signal processingpossibilities and control signal extraction for controllingsynthesizers, signal processing, lighting, and special effects. Thoseinstruments with sympathetic strings, such as the Esraj, Dilruba, andSarangi, may also include pickups for those strings as described inprevious example instrument adaptations.

Because each string has its audio charnel picked up intimately with thestring, it is possible to diminish some effects of the body resonanceand replace it with electronically created resonances. In particular,vocal sounds are known to appeal to the ear as vocal in nature due tothe relative center frequencies of a pair of predominant resonances asillustrated in FIG. 17. Through electronic synthesis of these resonancesthe vocal character of the instrument can be changed and, in fact,varied over time if one would dare to make the vocal bowed instrumentliterally sing.

Further, because of the somewhat different role of bowing, moreattention can be paid to collecting control information from the bow.However, the bow sensor techniques described can also be used to greatadvantage in Western bowed instruments. FIG. 42 shows a bow fitted withsensors to gather information from the hand, bow hairs, and bow motion.The bow 4200 fitted with bow hair 4201 as usual, may include a bow-hairtension sensor 4202, a free finger pressure-sensor and/or null/contacttouch-pad (or pressure-sensor array) 4203, a handle grip 4204pressure-sensor 4205, and an internal accelerometer 4206 to determinebow direction and acceleration. These control parameters can betransmitted to the instrument by attached electrical or fibre cable,wireless optical, wireless radio, or other means. As a simple example,the free finger controls the choice of resonant vowel formants for theinstrument while the hand grip pressure 4205 or pressure on the freefinger control 4203 may be used to control the sympathetic stringamplitude or a signal processing parameter. The bow tension andaccelerometer measurements can be used to control emphasis signalprocessing or darkening lowpass filtering.

4.9 Flutes and Recorders

Reed instrument layouts have been used in wind controller products byAkia and Yahama. However, flute-like (embouchure air hole) andrecorder-like (fingers normally down) instruments have to date not beused as models or methods for electronic instrument controllers.

It is noted that some types of Western flutes have at least some openholes, many folk and non-Western flutes have only open holes, and someflutes and recorders have at least one hole that is open but is operatedby a levered key. In the discussion below, the flute example isconsidered to be purely closed hole and key operated while the recorderexample is considered to be purely open hole without levered keys; thethus illustrated techniques can be freely applied to other hole andlever arrangements of a particular instrument variant. FIG. 43 showsadaptations of a Western (closed-hole) flute and an (open hole) recorderlayouts with pressure-sensors, or small pressure-sensor arraytouch-pads, replacing sites of the keys, together with air turbulencemeasurements, and air pressure average measurements as provided for inthe invention. In either instrument adaptation, the instrument may bemaintained as a sounding instrument or used as a model for an allelectric controller. In a sounding version, an attached or internaltransducer may be used to provide both an audio signal for processingand a means for note event control extraction (for example, usingpitch-to-MIDI technologies such as those of, or superseding, the RolandCP-40).

In the example flute and example recorder shown in FIG. 43, the bodyarea around the flute embouchure wind opening 4301 and recorder windopening 4351 may experience air pressure changes and turbulence whichmay be measured with sensors and signal processing as described earlier;if the sensors and wiring to them are mounted securely in low profilethe instrument behavior and sound will not be noticeably affected.Alternatively, in non-sounding versions, more obtrusive internal sensingof air turbulence and/or air pressure may be employed.

In a sounding adaptation of the closed hole flute 4300, the area of thekeys which contact the fingers 4302 can be covered with simple switches,a pressure-sensor, or a pressure-sensor array. Alternatively, in anon-sounding controller adaptation of the closed hole flute 4300, thearea of the keys which contact the fingers 4302 can be replaced bysimple switches, a pressure-sensor, or a pressure-sensor array. In thecase of the open hole recorder, simple switches, a pressure-sensor, or apressure-sensor array can be put around the perimeter of any of eachsingle-hole 4353, each double-hole 4354, and the thumb-hole 4355.Because of special playing techniques associated with the double-holes(i.e., “half-covering”) and thumb-hole (thumb tip flip or other“half-covering” methods), these areas may be handled with morespecialized switch and/or sensor arrangements.

For the most part such hole-positioned and key-positioned sensors may beused to assist in issuing note events but ranges of additional techniquecan be developed for more sophisticated control. A lesstechnique-oriented approach would be to put simple switches, apressure-sensor, or a pressure-sensor array in an area 4306, 4356 wherea thumb is otherwise only used for supporting the instrument.

As with the other instrument examples, it may also be advantageous toplace additional instrument elements such as strum-pads, touch-pads,sliders, switches, buttons, other sensors, etc., on the body of theinstrument.

4.10 Gongs, Bells, Cymbals, Chime Bars, other Metallaphones, andAcoustic Drum Heads

Gongs, bells, cymbals, chime bars, xylophones, and other metallophones,as well as the stretched heads of acoustic drums, can be problematic toamplify because they typically undergo significant displacement motionwhen struck yet their sound may alter significantly if this motion isrestrained and/or if a surface transducer is attached to them. It isnoted that there are many types of musically useful non-stereotypicalgongs with widely varying timbres, including for example thenon-crashing, pitched Indonesian gongs with close-set overtones whichbeat at low frequencies creating a complex tremolo effect that sounds inmany of these instruments very similar to pitch vibrato.

The invention provides for quality audio signal capture from these typesof instruments because of their musical usefulness, the richpossibilities for signal processing their sounds, and the visual appealof their playing in a performance situation. FIG. 44 shows how anoptical pickup may be created for a suspended gong; this technique mayalso be used for many other types of metallophones and acoustic drumheads. In the example, a gong 4400 is supported by small ropes 4401 athrough holes 4401 b or other means. Etched into, polished into, sprayedonto, adhered onto, etc. the gong 4400 is a reflective or refractivearea 4404 which may be unstructured or may include a reflective patternor refractive properties. One or more, typically two, light sources 4402a, 4402 b, typically coherent and/or modulated, direct light beams 4403a, 4403 b to the reflective or refractive area 4404. The materials,light sources, and geometry are arranged so that the return light path4405 is modulated in amplitude and/or intensity when the gong vibratesat audio frequencies. Reflective or refractive light 4405 from the gongis directed at one or more light detectors 4406 which use the resultinglight amplitude or intensity modulation to create an audio signal.Lowpass filters are used to remove subsonic signals resulting from theswaying of the gong. FIG. 45 shows an example metal bar which can bemounted so that extremes of impact displacement are relatively confined.In this arrangement the bar 4450 is suspended by small ropes 4451 athrough holes 4451 b or other means. The vibration of the bar may becaptured in a number of ways; illustrated are a similar optical sensor4452, and electromagnetic pickup 4453 should the bar 4450 beferromagnetic or have a ferromagnetic plating or attachment near thepickup, and a capacitive plate 4454 which can be used to measure thevarying capacitance between the plate 4454 and the bar 4450 via wires4455 a, 4455 b connected to the plate 4454 and a conducting support rope4451 a or other means. Similar methods may be employed for acoustic drumheads. Not shown is the case where a bar or tyne is tightly secured onone end; in this case not only may the bar be amplified by similarmethods but also by a piezo sensor in the support as in the case of theMbira discussed earlier.

It is noted that these pickup strategies all pick up localizedvibrations from the metallophone. As with instrument strings, theproduced timbre will vary widely with the selected pickup area. It istherefore provided for in the invention that multiple pickup areas maybe used, permitting multi-channel signal processing to be applied to asingle gong in a way like that described earlier for instrument strings.

FIG. 45 shows example gong arrays as part of a one-hand or two-handpercussion instrument stand 4500. The top tier of gongs 4501.1-4501.n inthis example are shapely gongs such as the Indonesian gongs describedearlier. The middle tier of gongs 4502.1-4502.m in this example are flatgong plates such as those found in Chinese percussion. Also shown inthis example instrument are two sets of acoustic drums 4603 configuredfor one-hand rolls as described earlier for electronic pads; thevibration of the heads of these drums are captured as per the describedmethods for gongs and bars.

5 Alternative Audio and Control Signal Sources

Historically new instruments have been created through incorporation ofnot only newly developed technologies but also newly discoveredphenomena. In this section recently available understanding of largelyunrecognized or unutilized processes are adapted by the invention foruse in generation audio and/or control signals.

5.1 Chemical Oscillations, Patterns, Waves, and Rhythms

The Belousov-Zhabotinskii reaction [Tyson] and many similar“non-equilibrium” chemical reactions exhibit oscillatory and animatedpattern-forming wave propagation and mathematical chaos effects whichcan be visually and electrically monitored [Gray, Scott]. Thesebehaviors are the result of nonlinear dynamics governing the evolvingreactant concentrations varying within the mixture over time [Nicolis].Varying types of electrodes can be used to measure component reactantsindependently. If multiple electrodes are used, differing but correlatedwaveforms are produced simultaneously, useful for both control andspatial timbre formation methods described later on. To some extentthese reaction processes may also be controlled [Ruoff; Nagy-Ungvaraiet. al.] via electric fields, reactant modulation, etc.—means that infact can be controlled directly or indirectly by electrical signals.Chemical indicators may be used to visually enhance the observablecontrast of pattern animation [Tyson; Orban et. al.]. The resultinganimated patterns, which range from swirling spirals to complex tidalforms—remnants of 1960's animated hallucinogenic iconography—can becaptured by video camera. The character of the patterns have visual andintuitive appeal and familiarity because they readily occur in biology,geology, and other parts of nature [Nicolis, Baras]. Populations ofthese chemical systems can be coupled by various means and as thus areobserved to have rhythmic and turbulent behaviors [Kuramoto]. Thesevarious dynamical properties of non-equilibrium chemical reactions canbe adapted to create a new exciting class of instrument entities andperforming environments which are described herein.

5.1.1 Chemical Oscillators as Sound Sources

In their simplest form, these chemical reactions act largely as simpleoscillators [Tyson; Gray, Scott]. The oscillations are the result ofnonlinear dynamics governing the evolving reactant concentrationsvarying within the mixture over time and typically are in the form ofslowly evolving limit cycles [Field, Noyes; Gray, Scott]. Eachreactant-monitoring electrode then produces an oscillatory signal forthe duration of the oscillatory concentration variation of thatreactant.

In practice most oscillations occur at very slow rates, for example witha period of 40-60 seconds, and have a short life time, for example undera hundred cycles, unless reactants are refreshed. The design of widerranges of chemical oscillators has been investigated [Epstein; Epstein]and in that it is conjectured that chemical oscillations may driveinsect wing vibrations it may be possible to design triggered chemicaloscillators that oscillate at audio rates with various oscillatorydurations. Such chemical reactions, when electrically monitored, can beused directly as sound sources in the same manner as anelectromagnetically-monitored or piezo-mechanically-monitored guitarstring.

Less speculatively, recorded measurements of known slow short-livedoscillatory chemical reactions [Gray, Scott] may be captured andprocessed as “audio samples” which can be pitch-shifted and spliced forarbitrary duration with conventional audio sampling technology. Further,mathematical models of these oscillatory behaviors [Field, Noyes] can benumerically simulated and altered so as to change rate, duration, andother attributes [Wang, Nicolis] as per model-based audio synthesis.Such numerical models then add a new non-acoustic class of modeledelements to the well establish acoustically vibrating ones such asstrings, pipes, tynes, membranes, etc., and as with the acoustic-basedmodels, can be adapted and extended to create yet other new effects.

5.1.2 Chemical Patterns as a Dynamic Controller

The inherent time scales of visual and electrically measurable patternevolution in most of the well-know non-equilibrium chemical reactions,along with their potential for direct and indirect electricalcontrollability, makes these non-equilibrium chemical reactionsinteresting candidates for the generation of control signals. Theinvention provides for the spatial patterns of these non-equilibriumchemical reactions to be measured and converted into control signals andpotentially, with any of several chemical processes, to control viacontrol signals and/or to video capture for display or recording. Theinvention provides that these measured control signals may be used tocontrol any one or more of note events, signal processing, lighting, orspecial effects.

The invention provides for spatial patterns of these non-equilibriumchemical reactions to be measured electronically by specific types ofelectrodes [Gray, Scott] and/or via a video camera combined with imageanalysis, parameter extraction, and control signal assignment. Ifelectrodes are used, these may be of various types, including thoseresponsive to variations in specific families of ion concentrations[Gray, Scott] as well as those used to measure electric fields,potential differences, electrical resistance, etc. These electrodes maybe distributed spatially in one, two, or three dimensions. FIG. 46illustrates spatial arrays of electrodes which may be used formeasurement, as well as control, in two-dimensional andthree-dimensional configurations. In an example two-dimensionalconfiguration, a shallow vessel 4610 has its bottom surface fitted witha two-dimensional array of electrodes 4611 selected from among the typescited earlier. These electrodes connect by wires 4612 to interfaceelectronics 4613 that extracts the measurement information, from thisextracts parameters (potentially under stored program control), and thenassigns these parameters (potentially under stored program control) tooutgoing control signals 4614. Some of the electrodes 4611 may bemeasurement only, some may be control only, and some may be dual-use.Any control capabilities and/or stored program recall may be controlledby incoming control signals 4615. In an example three-dimensionalconfiguration, a volume vessel 4620 has its bottom surface fitted with athree-dimensional array of electrodes 4621 selected from among the typescited earlier. These electrodes connect by wires 4622 to interfaceelectronics 4623 that extracts the measurement information, from thisextracts parameters (potentially under stored program control), and thenassigns these parameters (potentially under stored program control) tooutgoing control signals 4624. Some of the electrodes 4621 may bemeasurement only, some may be control only, and some may be dual-use.Any control capabilities and/or stored program recall may be controlledby incoming control signals 4625. Also illustrated in thethree-dimensional example are an example reactant intake tube 4632 andexample reactant outtake tube 4634 through which new reactants 4631 maybe introduced and old reactants 4633 may be extracted; this arrangementcan be used to vary the relative concentrations of chemicals in thevessel or maintain balance through reactant refreshing. The exchangerates and relative concentrations levels can be electrically controlledby conventional chemistry instrumentation means and thus the inventionprovides for these to be potentially controlled by incoming controlsignals. It is understood that multiple intakes and outtakes such as4632, 4634 may be utilized, and that the intakes and outtakes may haveinterior spatial distribution structures within the vessel 4620. It isalso understood that these methods would also be applicable forone-dimensional or two-dimensional arrangements and other vessel shapessuch as dishes, tubes, etc.

It is also possible to measure the evolving chemical patterns with avideo camera, particularly when differentiating visual indicatorcompounds [Tyson; Orban et. al.] are introduced into the mixture. FIG.47 shows an arrangement where evolving chemical patterns 4700 in thedish 4610 of FIG. 46 are illuminated with light sources 4701 andvisually monitored by an overhead camera 4702; here the light sourcesand camera are supported by arbitrary stand methods 4703. The inventionprovides in particular for the video camera 4702 to be combined withsimple to complex real-time image processing operations such as positionthreshold detection, edge detection location, pattern detection, patternarea measurement, pattern rate of change, etc. to derive a multitude ofparameters which may be mapped into control signals. Because visualchemical indicators can be used to identify localized concentrationslevels of specific chemicals, the aforementioned pattern recognitiontechniques may in some configurations conceptually be used in place ofthe potentially more costly chemical-measurement electrode arrays andsupporting electronics.

The invention provides for the aforementioned arrangements to be used asan interactive chemical performance environment. Outgoing controlsignals generated by the spatial chemical patterns may be used tocontrol any one or more of note events, timbre modulation, lighting, andspecial effects. Incoming control signals provided by or extracted fromaudio signals, electronic instrument elements, real-time sequencers,actuators, video cameras, or body position indicators (gestures, dance,stage position) can be used to control the evolution and influence theshapes of the chemical patterns. Video of the patterns may be displayedon monitors or projected, via video projector, onto the stage areabehind, above, or on one or more performers. The projected video imagemay be actual or processed by video signal processing (for example,changing color maps, contrast, solarization quantization thresholds,etc.) which in turn may be controlled by control signals generated inreal-time by the performers. In this manner, one or more performers mayinteractively perform with music, sound, and visual effect with anon-equilibrium chemical reaction environment.

It is also possible to numerically or electronically simulate thechemical dynamics on a computer, generating similar types of controlsignals, visual output, and interactive control capabilities. Thismethodology is discussed in more detail. Because numerical andelectronic simulation can generalize the process beyond physicallimitations, in principal a broader range of interactive dynamics wouldbe made possible by this method. However, the excitement and charm ofinteracting with a live chemical process is difficult to entirelyreplace with a computer program.

5.2 Photoacoustic Sources

Photoacoustic phenomena is a relatively new area of study. Although mostof the gathered knowledge and work in progress is largely oriented toprobably inapplicable areas relating to, for example, non-destructivetesting, there are a few phenomena, such as light stimulated acousticemissions and the modulation of light through vibrating transparent ortranslucent materials that can be developed for musical purposes[Lusher; Murphy et. al.; Bicanic, Dane]. The invention provides for theincorporation of these, particularly in that light can be used as partof performance and visually recorded material.

FIG. 48 illustrates example optical measurements of photoacousticphenomena in applicable materials 4900, 4910 which may be converted toelectrical signals, and an example electro-acoustic measurement ofphoto-induced acoustic phenomena in applicable materials 4920. Examplephotoacoustic materials 4900, 4910, 4920 may be in gas, crystal, liquidcrystal, plastic, elastic, fluidic or other forms.

For material 4900 which emits light in response to acoustic vibration, alight sensor 4901 may be used to recover the light emission event. Formaterial 4910 which modulates light in response to acoustic vibration, alight sensor 4901 may be used to recover light provided by a lightsource 4902 which is directed through the material 4910. For material4920 which emits acoustic vibration in response to light, anelectro-acoustic sensor 4921 may be used to sense acoustic vibrationemitted in response to one or more appropriately positioned lightsources 4902 a, 4902 b.

In the above, it is noted that ultra-sonic vibration, even up to a fewhundred Khz, is still potentially useful as these signals may bepitch-shifted or heterodyned down to audio ranges.

5.3 Electronic/Numerical Dynamical System and Relational SystemSimulation

Electronic and/or numerical algorithm methods may be used to implementmathematical dynamical models including mechanical vibration, fluidmechanics, stellar evolution, biological processes, etc. as well asabstract non-equilibrium, fractal, and chaos process models. Suchmethods are already in place in the synthesis of musical sound vibrationprocesses modeling conventional musical instruments, for example, inmodel-based sound synthesis as used in the Yahama VL1.

Because numerical and electronic simulation can generalize the processbeyond physical limitations, in principal a broader range of interactivedynamics and real-time measurements of them would be possible ascompared to that which could be obtained in the real-world underrealistic conditions. Further, electronically or numerically modeledprocesses may be time-scaled so as to produce audio frequencies or moreslowly evolving control signals. The invention provides for the use ofsuch electronic and/or numerical algorithm methods so as to implementmathematical dynamical models of adapted real-world or abstractprocesses. Incoming control signals can be used to select and/or affectthe structure and/or parameters of the modeled dynamics and/orrelations, and the modeled dynamics and/or relations may be used tocreate any one or more slowly varying outgoing control signals, visualimage signals, or direct audio signals.

Examples of abstract processes may include interactive navigationthrough a fractal structure, the fractional integration of an audiofrequency square wave as it evolves into triangle and parabolicwaveforms, etc. Examples of real-world models rich in semiotic value forperformance may include adaptations of interactive control of galacticinteraction dynamics, language models, etc. as well as the use ofliterary plots, classical mythologies, etc. which have been used bycomposers for centuries (i.e., Monteverdi's Orfeo, Strauss' Electra,Stravinsky's Odepus Rex, etc.)

5.4 Environmental

Earlier instrument elements and instrument entities associated withenvironmental aspects of stages, rooms, and the outdoors were described.Examples of this include the tracking of the position and/or motion ofperformers, the tracking of artificial fog cloud migration, roominternal and outdoor meteorology, and audience motion activity. Asindicated in those discussions, these may be used, to the extentartistically applicable, to generate control signals for the control ofnote events, signal processing, lighting, and special effects.

6 Generalized Instrument Interfaces

Referring to FIG. 1, it is recalled that the invention provides for bothinstrument entities 100 and signal routing, processing, and synthesisentities 120 to be fitted with compatible electrical interfaces, termedgeneralized instrument interfaces or (or more concisely,generalized-interfaces) 110, which can exchange any of the following:

-   -   incoming electrical power (111)    -   outgoing control signals from switches, controls, keyboards,        sensors, etc., typically in the form of MIDI messages but which        may also include contact closure or other formats (112)    -   control signals to lights, pyrotechnics, or other special effect        elements within and/or attached to the instruments, said signals        being either in the form of MIDI messages, contact closure, or        other formats (113)    -   outgoing audio signals from individual audio-frequency elements        or groups of audio-frequency elements within the instruments        (114)    -   incoming excitation signals directed to individual        audio-frequency elements or groups of audio-frequency elements        within the instruments (115)    -   outgoing video signals (such as NTSC, PAL, SECAM) or image        signals sent from the instrument (116)    -   incoming video signals (such as NTSC, PAL, SECAM) or image        signals sent to the instrument for purposes such as display or        as part of a visually controlled instrument (117).

The interfaces may be realized by any one or more of connectors, cables,fibers, radio links, wireless optical links, etc., individually, incombinations, or in or sequences of these.

In most envisioned realizations this interface would be involve one ormore connectors fitted with driving and/or receiving electronics, andthe connectors on instrument entities 100 and signal routing,processing, and synthesis entities 120 would be connected by a pluralityof wires in either balanced or unbalanced transmission mode.Alternatively one or more coax cables, fiber optic cables, radio links,wireless optical links, etc. may be used to replace part or all of theplurality of wires. Any of these approaches may use any of a variety ofmultiplexing techniques [frequency-modulated and/or phase-modulatedand/or amplitude-modulated carrier, wavelength-division, time-division,carrier-less constellation synthesis (such as CAP), statistical, etc.)individually or in combination to reduce the number of partitionedphysical signal channels (wires, fibers, radio channels, wavelengths,etc.].

When these generalized interfaces are realized via one or more physicalcables (electrical, optical, etc.), some realizations may use a singleconnector for fully spanning generalized applications while otherrealizations may consist of an ensemble of connectors in a functionalsplit so as to handle particular organization, expansion, and/orevolutionary needs.

FIG. 49 shows how generalized interfaces can be built in whole or viaseparable parts which may be used selectively as needed or appropriate.In a single connector all-purpose approach, a single cable 4901 carriesall the signals allowed for in the generalized interface up to apredefined number of maximum instances. Referring to FIG. 1, allinstrument entities 100 and signal routing, processing, and synthesisentities 120 would provide mating connectors to that of 4902. In oneimplementation, the cable 4901 and connector 4902 provide a relativelylarge number of separate interconnecting wires, for example twenty tothirty. Alternatively, in a future preferred arrangement employing thenlow-cost standard signal multiplexing and/or directional multiplexingtechniques, this cable 4901 could be a simple one or two fiber optic orcoax cables potentially supplemented with two to three power conductors;alternatively, if any coax cables are used to carry signals they canalso be used to simultaneously carry power on the same conductors.Further, if each instrument is able to provide its own power by means ofwall plugs and/or batteries, the aforementioned implementations need notinclude any power carrying capabilities and associated conductors. Inthis arrangement it will be further possible to use radio and/or opticalwireless channels to carry the signals among instrument entities 100 andsignal routing, processing, and synthesis entities 120. In thisarrangement then, a common multiplexed incoming and outgoing signal,potentially itself directionally multiplexed on the same channel, can becarried interchangeably by optical fiber, coax cable, wireless radio, orwireless optical transmission mediums.

Alternatively, it is possible to functionally partition the generalizedinterface into standardized component interfaces which may be served byseparate connectors. A multi-connector “Hydra” cable can be used toprovide selected groups of two or more of these standardized connectors,including a “fully-populated” Hydra cable with all the definedconnectors. If only one connector of the several defined ones is needed,then a single connector cable may be used if desired; for this reason,it may be desirable to assign connectors to the functional partitionswhich are standardly available on mass-produced single connector cables.FIG. 49 shows an example of this arrangement. A fully-populated Hydracable 4911 may include the following partition and assigned connectors:

-   -   Two “single-channel” group audio outputs (unbalanced, dedicated        ground) 4912; for example, a TRS male plug.    -   Ten or twelve multi-channel audio outs (unbalanced, dedicated        ground) and audio power 4913; for example a 13-pin DIN male or        HDB 15-pin VGA male (with 14 pins actually populated and        connector shells interconnected).    -   One video out channel and one video in channel (unbalanced,        dedicated ground) and incoming video power 4914; 6-pin DIN male.    -   One balanced MIDI out channel and incoming MIDI power (on        non-MIDI pins) 4915; 5-pin DIN male.    -   balanced MIDI in channel and incoming controlled element power        (on non-MIDI pins) 4916; 5-pin DIN male.    -   Six excitation drive channels (unbalanced, dedicated ground)        4917; 8-pin DIN connector (only six pins need if the signal        routing, processing, and synthesis entity 120 creates the        relatively high-power drive signals, while eight pins allow        provision of power to the instrument entity 100 for internal        high-power signal generation).

Further, any connectors not served by a given Hydra cable and/orexpansions to support additional channel-carrying needs may be supportedwith additional cables:

It is understood that the aforementioned as explained and illustrated inFIG. 49 are merely examples; other arrangements are possible.

7 Signal Routing, Processing, and Synthesis

The general principals for the architecture of the signal routing,processing, and synthesis entity 120 as provided for by the inventioninclude all or a significant number of the following:

-   -   flexible multi-channel handling of audio, control, and video        signals    -   a hierarchically modular control and control signal routing        structure    -   course to very fine-grain control signal routing (for example,        in the context of MIDI, routing at the MIDI port level, routing        the MIDI channel level, and routing at the individual note        number and continuous controller number levels)    -   the incorporation of mixing in audio routing and message merging        and polyadic operations in control signal routing    -   control signal extraction from audio and/or video    -   audio signal, control signal, and potentially video signal        processing    -   audio signal, control signal, and potentially video signal        synthesis    -   real-time control signal/event replay        all under extensive real-time control.        7.1 Audio Signal Routing

Audio signal routing is provided for in the invention by both switchingand mixing functions. Switching functions may be realized as storedprogram cross-bar switches. Mixing functions may be provided in the formof possible multiple-input multiple-output mixing matrices and anadditional final mixing stage may include some dedicated signalprocessing functions. Mixing functions provided for in the invention areadvantageously controlled in real-time by control signals.

Functional examples of the functionality provided for in saidmultiple-input, multiple output mixing matrices is that of the SoundSculpture model Switchblade MIDI-controlled mixer (but empowered with asignificantly larger number than two MIDI continuous controller inputs)or the Peavey PM-8128 (but provide with additional inputs and outputs).Functional examples of the functionality provided for in said finalmixing is that of the Yahama DMP MIDI-controlled mixer models,particularly the DMP 9-16 (but with additional presets). In theinvention, the mixing and switching functions are preferably an within alarger-scale hardware and software construct rather than anoff-the-shelf module.

7.1.1 General Audio Switching and Mixing

Referring to FIGS. 1-2, input signals directed to audio routing andmixing may include the audio outputs from the instrument entities 100,outputs from audio signal processing elements 125, and audio signalsynthesis elements 129 a. Still referring to FIGS. 1-2, output signalsdirected from audio routing and mixing may include audio inputs toexciter elements within instrument entities 100, audio signal processing125, control signal extraction 128 a, and the overall audio outputs toamplification and/or recording facilities.

7.1.2 Multi-Channel Audio Signal Handling

The invention provides for extensive support for and exploitation ofmulti-channel audio signals from instruments with multiple vibratingelements.

Multi-channel transducers have been used in multiple-vibrating-elementmusical instruments; these uses appear to be confined to guitarsynthesizer interfaces (as with the Boss GP-10), individual adjustmentof each vibrating element mix level (as with the Gibson Chet Akinsguitar), and creation of panned stereo mixes. (Biax pickup, Passaicsynthesizer interface, Turner string pan-pot guitar, Van Halen-endorsedguitar with right/left switches for each string). These similarapproaches may be generalized by a common diagram. FIG. 50 showsmultiple vibrational elements with multi-channel transducers applieddirectly to stereo or multi-channel mix-down. Each of a plurality ofvibrating elements 5001.1-5001.n are is coupled by appropriate butseparable means to each of a plurality of transducers 5002.1-5002.n,each producing electrical signals which are applied to either singlechannel or stereo mixing circuitry. In the case of the Gibson ChetAtkins electric-acoustic guitar, the mix is monaural and as such isequivalent to delivery only through output A 5004 a, with noimplementation of output B 5004 b. In the case of the Biax pickup, the“pan” position in the stereo field is hard-set to either output A 5004 aor output B 5004 b. In the case of the Turner guitars, the “pan”position in the stereo field is set between output A 5004 a or output B5004 b via adjustable potentiometers.

Specifically the invention provides for bringing the signals frommulti-channel transducers 5002.1-5002.n to individual signal processingstages 5005.1-5005.n before mixing, allowing far more extensivecapabilities to be created. FIG. 51 shows multiple vibrational elementswith multi-channel transducers and individual signal processing prior tomixing. FIG. 51 highlights the functional signal processing distinctionfrom existing commercial products generalized by FIG. 50.

This relatively simple conceptual (though potentially hardware and/orsoftware intensive) change makes a number of extraordinary thingspossible:

-   -   Conventional pitch-shifting signal processing can be used on        each string signal to create:    -   “generalized pedal steel guitars” (augmenting or replacing        mechanical pedal tuning changers with pedal, lever,        spring-wheel, optical, or other electronic controls determining        pitch shift amount)    -   instantly retunable guitars (augmenting or replacing mechanical        tuning changers such as the Hip-shot “Trilogy”)    -   a true electronic simulation of so-called “multi-course”        instruments (such as a 12-string guitar, mandolin, lute, etc.)        where individual elements making up the “multi-course” are        simulated using pitch shifting to create either octaves or        slightly mis-tuned unisons    -   multi-key, multi-modal Indian sitars; here drone and sympathetic        strings can be electronically retuned while playing, allowing a        more flexible and robust mix between Eastern (fixed tonality)        and Western (modulating tonality) musical forms.    -   multi-key, multi-modal African mbiras, African koras, Japanese        kotos etc.; here fixed pitch vibrating elements (tynes, strings)        can be electronically retuned while playing, allowing a more        flexible and robust mix between Eastern (fixed tonality) and        Western (modulating tonality) musical forms.    -   spatial-spectral animated instruments where individual vibrating        element sounds may be location modulated within a stereophonic        or other spatial sound field (using low-frequency sweep        chorusing, continuous auto-panning, etc.).    -   separate distortion circuits for each vibrating element, for        example, to create: powerful guitar chord sounds previously        obtained only by multiple instruments or multi-track recording)    -   simulated sitar-bridge effects (using the methods to be        described in conjunction with FIG. 39)    -   finely frequency-equalized instruments where different frequency        equalizations are applied to each vibrating element.    -   mixed timbre instruments where different signal processing        methods are applied to each vibrating element.

(More recently, a functionally limited—although very technologicallyprogressive—version of the signal processing approach illustrated inFIG. 52 has since appeared in the Roland VG-1 COSM guitar emulationproduct.)

FIG. 52 shows addition of a control signal extraction element to thearrangement of FIG. 51. As shown in FIG. 52, the invention provides forsignal outputs from the multi-channel transducer(s) to be fed inparallel to a control signal extractor 5010 so as to simultaneouslyissue control signals 5011 to any one or more of music synthesizers,signal processors, lighting, special effects, etc. The invention thereinprovides for an operating mode of control signal extraction where eachindividual vibrating element to have associated with it one or moreindividual control signals. The resulting system, then, can assign agiven vibrating element to individual signal processing, individualsynthesizer voice, or both in combination. This permits a basicconventual instrument structure (such as a guitar, violin, steel guitar,koto, or mbira) and essentially conventional playing techniques tocontrol an unprecedented rich range of sounds. (Even more recently,Roland has since announced a cable fan-out product allowing their VG-1multi-channel signal processor to operate simultaneously with theirGP-10 and related guitar to MIDI interface products.)

Further enhancements are also possible. For example,

-   -   in practice it may be desirable to have a different number of        signal processors than vibrating elements; for example:    -   in generalized steel guitars, only a few strings at a time may        actually be candidates for pitch shifting    -   in spatial-spectral (panning, chorusing, etc.) animation, the        actual number of animation channels need not match the number of        vibrating elements    -   in a highly functional system, several signal processors may be        used in parallel for one or more vibrating elements.

FIG. 53 shows partial mix-downs of vibrating element signals fed to anumber of signal processors 5005.1-5005.n and straight-through paths5007.1-5007.m en route to subsequent mix-down. The (pre-signalprocessing) input mixer 5006 is used to route and/or mix variousmulti-channel transducer signals to a structured and controllablemulti-channel mix. Several variations of this arrangement are suggestedby FIG. 53:

-   -   there can be zero, one or more straight paths 5007.1-5007.m    -   there are a total of at least three signal processors        5005.1-5005.n and/or straight paths 5007.1-5007.m involved    -   there is a minimum of one signal processor (or else the two        mixers 5003 and 5006 functionally collapse into one,        functionally resulting in the arrangement of FIG. 50)    -   although the interconnection details for connecting the        synthesizer interface are shown, the synthesizer interface need        not be included.

FIG. 54 shows another arrangement wherein a switching matrix 5008 isused in place of the input mixer 5008 to select which individualvibrating element 5001.1-5001.n signal is assigned to which signalprocessor 5005.1-5005.k. This arrangement is particularly relevant tothe generalized retuning of certain vibrational elements, as in adaptedpedal steel guitar and adapted sitar described earlier.

The invention also provides for several signal processors to be pooledand used in various parallel, series, or other topologicalinterconnections serving one or more vibrating elements. FIG. 55 shows amore flexible example method for providing signal processors withvibrating elements' signals and other signal processor outputs viaswitch matrix, and additional partial mix-downs by replacing said switchmatrix 5008 with an input mixer 5006.

The invention provides for any of the above systems to be integratedtogether into a common system sharing a common configuration presetstorage and recall facility. FIG. 56 shows configuration control ofsignal processors, mixers, any switch matrix, and synthesizer interfacesvia logic circuitry and/or microprocessing. As illustrated in FIG. 56,this is simply a matter of putting all or some combination of the mixers(5003 and/or 5006), switch matrices 5008, signal processors5005.1-5005.k, and/or synthesizer interfaces, as relevant, under thecontrol of logic circuitry and/or microprocessors 5009 which can providesuch preset storage and recall functions.

By combining the multi-channel signal handling with excitation, not onlycan individual vibrating elements be assigned to various signalprocessing and synthesizer controlling roles, but also individualvibrating elements can now be assigned feedback modes where selectedvibrating elements can sustain vibration as if they were bowed, in anelectric-guitar feedback arrangement, etc. Further, through use ofadditional switching, signal processing can be added to the feedbackloop as discussed earlier, but on an individual vibrational elementbasis. Finally, since feedback arrangements tend to emphasize higherharmonics of vibration, and the dynamics of the relative levels of theharmonic mix can be varied dramatically by touching elements or varyingfeedback characteristics (via signal processing in the excitationfeedback loop), the invention provides for control signal extraction tobe expanded to respond to details of the overtone content as discussedlater.

FIG. 57 shows a very general combined environment for multi-channelsignal processing, mixing, excitation, and program control of overallconfiguration. In FIG. 57 the general combined environment incorporatesa plurality of separate feedback loops for each vibrating element, eachloop featuring a loop signal processor 5021.1-5021.n (which here couldbe as simple as a level control) which may be controlled by controlsignals as provided for in the invention. Many possible variations ofthis approach which may omit or simplify any of the elements shown inFIG. 57 can be realized: for example the signal processors 5005.1-5005.1k and 5021.1-5021.n may be pooled together in their association withmixers 5006 and/or switches 5008 with said mixers 5006 and/or switches5008 sending signals to the excitation drive amplifiers 5022.1-5022.n.

7.2 Audio Signal Processing

Many of the audio signal processing elements cited as 125 (FIG. 1), 2211(FIGS. 25-26), 5005.1-5005.n/k (FIGS. 51-57), and elsewhere in thisdocument can be adequately realized by any number of the standardmulti-function MIDI-controlled signal processing modules such as theRoland model RSP-550, Boss model SE-70, Ensoniq model DP/4, ART modelSGE Mach-II, Digitech model GSP21, etc. In the invention, these signalprocessing functions are preferably an integrated component within alarger-scale hardware and software construct rather than anoff-the-shelf module. The invention provides not only for theincorporation of these into the signal routing, processing, andsynthesis entity 120 (FIGS. 1-2) in the ways described below but alsofor additional audio signal processing methods which are notcommercially available and which are described below.

7.2.1 Spatially Distributed Timbre Construction

Because of the extensive biaural capabilities of human hearing, stereoand other multi-channel sound fields can be used to create a number ofmusically useful timbral construction ranging from the subtle to thepowerful and the beautiful to the bombastic.

Examples of this, commonly found, are stereo-output chorus,stereo-output flangers, stereo-output reverb, stereo-output echos, etc.;but the spatial construction of timbres may be carried far beyond thesesimple and now commonplace effects. The following discussion explainssome example techniques; the role and value of these techniques aredeveloped further in subsequent material following that below.

7.2.1.1 Cross-Channel Modulated Delay

The invention provides for methods to enhance, and to more significantlyincrease the depth of, a stereo signal set source whose components havesimilar but slightly different timbres, particularly if the timbres aretime-varying. Examples of such stereo signal set sources include thestereo outputs of traditional choruses, flangers, reverbs, etc., a pairof signal distortion elements with different characteristics, twoharmonized synthesizer voices or pitch-shifter outputs, the separateoutputs of a single two-oscillator synthesizer voice, etc. FIG. 58 showsa stereo-input, stereo output configuration of two monaural flangeand/or chorus elements wherein the unaltered signal of each inputchannel is combined with a delay-modulated signal from the oppositechannel. Each input 5801, 5802 is presented to a dedicateddelay-modulation element 5809, 5810 and a dedicated output channelsummer 5831, 5832. Internally each delay-modulation element 5809, 5810consists of a variable delay implemented in this example by changing theclock speed of a clocked delay line 5803, 5804 by means of a variableclock oscillator 5805, 5806. The speed of the clocking oscillator iscontrolled by a variable low-speed modulating “sweep” oscillator 5807,5808. Since the sweep oscillator's output is periodic, it is possible toimplement the clocked delay line 5803, 5804 not only by changing thesample rate but by loading fixed-rate samples into a ring buffer andreading out of the ring buffer at a rate set by the variable clock 5805,5806. Other implementations of delay-modulation elements 5809, 5810 arealso possible as known to those familiar with the art. The outputs 5821,5822, respectively, of each delay-modulation element 5809, 5810respectively, are directed to the summing elements 5832, 5833,respectively associated with the opposite unaltered input channel 5802,5801, respectively, producing stereo outputs 5832, 5831. The delayedsignals 5821, 5822, may be summed at the summing elements 5832, 5833 inadditive, subtractive or other phase-shafted or phase-dispersedrelationships. In a preferred implementation the two sweep oscillators5807, 5808 operate at slightly different sweep frequencies and arefree-running (unsynchronized) if possible.

(It is noted that a similar, restricted version of this has since beenincorporated as one of the effector modes, namely “cross-over chorus”,of the Korg model X5DR synthesizer module. In the Korg implementation,however, the two sweep oscillators 5807, 5808 have been replaced by asingle sweep oscillator with two phase-locked quadrature, i.e.,90-degree phase difference, outputs.

It is noted the above arrangement may naturally be extended beyondstereo to accommodate additional input and/or output channels. The mostgeneral implementation would have N inputs, M outputs, N−1 variablespeed swept delays, and M summers with N inputs summed with adjustablegains and/or phase relationships; simplifications of course arepossible. One example application would include M-speaker (i.e., M=4 forquadraphonic) amplification. Another example application with M=2 forstereo and N>2 similar signal sources would build an enhanced version ofthe sonic effect.

In the above it is noted that when pluralities of elements (for example,spatializer and distortion elements) are cited, the elements in theplurality need not be identical in their type and/or parameterizedsettings. Further, various parameters of each of the elements(modulation speed, modulation depths, relative amplitudes in audiomixes, etc.) may be advantageously controlled in real-time by controlsignals for expression (from instrument entities, foot controllers,etc.), further correlation with the signal source (for example, usingenvelope extraction control signals) or further levels of animatedenhancement (employing additional sweep oscillators, envelopegenerators, etc.).

7.2.1.2 Multi-Level Stereo Chorused Distortion of Monaural Sources

The invention provides for creating a similar-signal stereo signal setfrom two distortion sources and presenting it to cross-channel modulateddelay to synergistically transform a relatively spectrally dull signal,particularly a time varying one, into a very rich powerful sound. FIG.59 illustrates a combination of a spatialized effect, two distortionelements, and a stereo (N=M=2) cross-channel modulated delay. In thisexample arrangement, the input signal 5900, which may be from a grouppickup as shown, individual vibrating element pickup, audio signalsynthesis element, microphone, etc., is applied to a stereo-outputspatializing effect 5901 such as a stereo output chorus, flanger, etc.The resulting stereo signals 5902, 5903 are applied to distortionelements 5904, 5905. Fluctuations in the waveshape of the applied stereosignals 5902, 5903 cause significant dynamic timbral shifts from thedistortion elements 5904, 5905; such fluctuations in the applied stereosignal waveshape may originate from the original signal 5900, from anyone or more of amplitude animation, phase animation, delay accumulation,dynamic waveshaping processes, etc. in the spatializer 5901, or both incombination. The distortion elements 5904, 5905 produce a richsimilar-signal stereo signal set 5906, 5907 which is then presented to aN=M=2 cross-channel modulated delay, resulting in stereo output signals5909 a, 5909 b.

For a single input channel 5900, the invention provides for theexpansion of such an arrangement to include additional processes tobuild an enhanced version of the sonic effect. For example, an N-outputversion of the spatializer 5901 (which may, for example, be implementedinternally by two or more simpler spatializers in parallel,hierarchical, or other interconnection topologies) can be used inconjunction with N distortion elements in an N-input (N>2) M-outputcross-channel modulated delay replacing 5908.

For multi-channel signal sources, the invention provides for each signalto be handled by a dedicated spatializer and several possible subsequentprocessing arrangements. As one example, assuming K input channels,selected outputs of each of the K spatializers may be mixed andpresented to N (N being two or more) distortion elements which in turnare presented to an N input, M output cross-channel modulated delayreplacing 5908. In another example, no pre-distortion mixing is used butrather each spatializer is provided with its own collection of two ormore distortion elements; the collection of all outputs of these, whichare of number J not equal to N, may be matrix-mixed to form N mixedoutputs which are applied to an N input, M output cross-channelmodulated delay replacing 5908. In another example, no pre-distortionmixing is used but rather each spatializer is provided with its owncollection of two or more distortion elements; the collection of alloutputs of these, which are of number N, may be directly applied to an Ninput, M output cross-channel modulated delay replacing 5908. Otherarrangements similar in form and spirit are clearly possible.

In the above it is noted that when pluralities of elements (for example,spatializer and distortion elements) are cited, the elements in theplurality need not be identical in their type and/or parameterizedsettings. Further, various parameters of each of the elements(modulation speed, modulation depths, relative amplitudes in audiomixes, distortion parameters, etc.) may be advantageously controlled inreal-time by control signals for expression (from instrument entities,foot controllers, etc.), further correlation with the signal source (forexample, using envelope extraction control signals) or further levels ofanimated enhancement (employing additional sweep oscillators, envelopegenerators, etc.).

7.2.1.3 Location Modulation

Location modulation has been commercially available in the form of“auto-panning” where an audio source is periodically panned back andforth between two stereo outputs. The invention provides for limitingperiodic auto-panning of monaural sources sounding in isolation to betypically most effective when the degree of panning is limited and themodulation rate is low (as extreme settings of modulation depth andspeed are typically not as widely musically useful). Under theseconditions in a stereo sound field a signal source takes on an animatedcharacter but yet is not so blatantly spectrally modified as it is inchorus and flanging effects. The invention also provides for widerranges of depth and speed to be used in the context of multi-channelauto-panning, discussed next, and layered signal processing discussedbelow and already touched upon in the discussion associated with FIG.39.

The invention provides for multi-channel versions of auto-panning. Inlayered signal processing, such as that discussed in the context of FIG.39, auto-panning contributions work best within the invention ifmodulation sweep oscillators operate at different (typically onlyslightly different) modulation speeds. In the context of multi-channelsignals provided by individual vibrating elements of an instrumententity 100, this unsynchronized configuration of modulation sweeposcillators can create inhomogeneous “bunching” effects when manymodulation sweep oscillators take on identical or nearly-identicalvalues in extreme ranges of the modulation sweep.

The invention provides for a much more homogeneous method formulti-channel periodic-sweep auto-panning, namely that of arranging thesignal pan images in a phase-staggered constellation swept by a singlemodulating sweep oscillator. A simple example is that of stereocross-panning where two input signals pan between stereo speakers insynchronized complementary directions. Another example is that ofstaggering the phases of a multiple phase output modulating sweeposcillator in some preassigned arrangement, such as offset from eachother by a common phase-offset value. This may be used to pan the soundsfrom each individual vibrating element so that the individual pannedsound images follow one another between two speakers. Similar methodscan be used if there are more outputs (for example, quadraphonic,hexaphonic, octaphonic etc. speaker installations aligned in a plane orin 3 dimensions); here N input, M output mixers can be controlled by oneor more single or multiple-phase output modulating sweep oscillators.

Control-signal invoked transient “one-shot” panning effects may also beobtained from commercial mixer products that feature a fade-timetransient between pre-programmed amplitude settings (such as the YahamaDMP series and Sound Sculpture Switchblade series). The inventionprovides for such transient effects to be used as a compositionalelement in music or a metaphorical or semiotic element in audio and/oraudio-visual aspects of performance. In particular limited-durationpanning trajectories of arbitrary nature, each affiliated with one ofseveral individual sound sources, may be made to simultaneously and orsequentially follow a predefined relative dynamical pattern. This can beused as a contrapuntal element in melody or abstract musical forms. Itcan also be used to create plot events in a composition or performance,such as in a musical composition, dance composition, or play concerningor involving the spatial interaction of bird sounds.

7.2.1.4 Other Spatially Distributed Timbre Methods

Several other aspects of the invention to be presented below in othercontexts also may be used to create spatially-distributed timbralrealizations; their use as general audio signal processing elements 129a in this fashion is provided for as part of the invention.

One aspect of the invention which may be used for spatially-distributedtimbral realizations is the two-input or multiple-input versions of theoctave cross-product chain described later on in the context of audiosignal synthesis waveshaping. As described there, this technique resultsin a number of parallel signal outputs with widely differing spectralcontents and spectral animation features, and the animation featuresslow to a halt when all fundamental and overtone frequencies of the twoinput signals are brought into fixed integer and small integer-ratiomultiplicative relationships. The aforementioned characteristics of themultiple outputs lend themselves to spatially-distributed timbralrealizations since mixing of the outputs can partition the frequencycontent and animation features differently between final mix-downoutputs. The invention provides for this method to be used as a signalprocessing technique. In one example usage, a pitch-shifter, sweptvariable delay, etc. is used to construct a derivative frequency and/orphase shifted signal (the characteristics of which may be controlled bycontrol signals for expression) from an original signal. The originaland derivative signals are then fed into the octave cross-product chainto produce often spectacular spatially-distributed timbral realizations.

Another aspect of the invention which may be used forspatially-distributed timbral realizations is multi-channel waveshapingwhere a signal source is provided to a plurality of waveshapers each ofwhich may be controlled by control signals. Each waveshaper output maythen have differing frequency content and animation features which thuslend themselves to spatially-distributed timbral realizations in amulti-channel (stereo, quadraphonic, etc.) partition or mix-down. Ofparticular interest is the use of hysteretic waveshaping, describedlater, which creates a wide range of spectral differences as the inputwaveform and/or hysteresis parameters change over time.

Another aspect of the invention which may be used forspatially-distributed timbral realizations is the use of later describedlayered audio signal processing methods. The invention does this byproviding for each audio signal processing layer to be allocated adifferent proportion to each final mix-down output channel. Theseallocated mix proportions may be varied over time by control signals.

7.2.2 Multi-Channel Audio Signal Handling

The invention provides for flexible homogeneous and inhomogeneous signalprocessing of multi-channel audio sources. Such multi-channel audiosources may for example include, referring to FIGS. 1-2, a multiplevibrating element instrument entity 100, multiple instances of audiosignal synthesis elements 129 a with single or multiple output-channelsaudio signal synthesis elements.

Several signal processing methods involving multi-channel signal sourceshave already been discussed thus far, particularly those in the previousfew sub-sections. The invention further explicitly provides fordedicated, shared, or combined arrangements for audio signal processingelements within the signal routing, processing, and synthesis entities100 as shown in FIGS. 53-57. In particular, each signal from amulti-channel source may be handled differently with at least some ofthese signals processed by one or more audio signal processing elements129 a of identical, similar, or differing function. Conceptually, themost flexible of these are embodiments where input mixing 5006 is usedto share the inputs and outputs of a pooled collection of signalprocessors as in FIGS. 55-57. The audio signal processing elements 129 amay be any one or more of conventional signal processing functionsprovided by commercial products (chorus, flange, reverb, distortion,delay, filtering, equalization, etc., individually or in combination) aswell as any one or more of the invention's novel audio signal processingmethods described thus far and below.

7.2.3 Bass Note Derivation

The invention provides for the derivation of bass notes from signalsources. This is particularly relevant in the invention where signalsfrom selected vibrating elements are used to create bass notes. Thecreated bass notes may be heard in parallel with the original pitch ofthe signal (each pitch may be subject to different signal processing) orin replacement of it. In many cases this completely eliminates the needfor bass accompaniment in a performance situation at the potentialexpense of melodic freedom of the bass line.

The invention provides for at least three methods of bass notederivation which may be used individually or in combination.

One of these methods is the use of control signal extraction to derivenote events to run a bass note audio synthesis element (for example, aconventional audio synthesizer module transposed down one or moreoctaves or other large interval). If the bass interval is not always tobe fixed, pre-programmed note transpositions reflecting desired harmonyand/or player-controlled changes in pitch-shift interval may be usedindividually or in combination. This audio synthesizer method allows awide range of sounds to be used but can be limited in how the bass noteexpression can be controlled from the original signal source. Onesolution to this provided for by the invention is the use of overtoneparameter tracking in the control signal extraction; these additionalparameters may be used to shape the synthesized sound though varyingparameters in the synthesis processes and/or by varying subsequencesignal processing parameters.

Another of the methods is through the use of conventional pitchshifters. If the bass interval is not always to be fixed, so called“intelligent-harmony” pitch shifters (such as the Digitech model IP-33B)and or player-controlled changes in pitch-shift interval may be usedindividually or in combination. The use of pitch-shifting allows fornuances of the original signal source to be carried through but maysuffer from delayed response, glitch, phasing, “Darth Varder,” or otherundesirable or limiting artifacts.

Yet another method, should the bass interval always be related to thesource pitch by octaves, the invention provides for an adaptation of thenovel octave divide method used in the Boss OC-2 “Octaver” pedal.Although this technology does have glitching and monophonic limitationsas described below, it works very well in responding to amplitudeenvelope attributes of the signal source. As is evident from thepublicly available published service note schematic and usage of thedevice, each octave signal is created by frequency dividing the originalsignal (for example by means of a toggle flip-flop), scaling itsamplitude by the instantaneous amplitude of the source signal (forexample, through use of an envelope follower and a gain-control method),and combining this with a bit of the original signal to create a richerresulting overtone result. The unit suffers from the fact thatharmonically rich signals often confuse the frequency dividers resultingin a very glitchy bass signal. Further, the method is monophonic; theplaying of two notes at once processes only one bass signal, and usuallyan unusably unstable one. The invention provides for the glitch-freeadaptation of the OC-2 technology to multiple vibrating elementinstruments by dedicating a specific low-pass filter and an allocated(or allocatable) OC-2 divider or divider chain to each selectedvibrating element. In particular, the incoming individual vibratingelement signal is low-pass filtered to greatly attenuate frequenciesabove the maximal fundamental frequency to be recognized by thearrangement (this maximal value may, in some circumstance be high enoughto support unfretted string “chime” harmonics and the like). Thecombination of applying each instance OC-2 technology to a singlevibrating element together with a highly emphasized fundamentalfrequency eliminates the glitching and monophonic limitations. Theinvention provides for a plurality of the described OC-2/filterarrangements, numbering for example three for a guitar, to be allocatedto specific vibrating elements (fixed by design, selectable via storedprogram control, etc.). Further, the invention provides for the use ofthis technology should bass notes need to be non-octave in relation tothe original signal: the nearest octave note can be generated by theOC-2/filter approach and an allocated pitch shifter may be used to makerelatively smaller pitch changes, recognizing that smaller shiftintervals tend to have less artifacts.

7.2.4 Layered Audio Signal Processing

The invention provides for the layering of multiple audio signalprocessing paths driven from one or more shared sources and partitionedor mixed down to two or more output channels. Because this may be viewedas a superposition of several signal processing paths, this will bereferred to as “layered audio signal processing.” One example of thishas already been presented in the discussion relating to FIG. 39; hereeach layer is responsible for emulating a separate sympathetic stringeffect. As each layer is in the FIG. 39 example essentially identical,the layers may be called “homogeneous.” (Some examples of homogenoussignal processing have since been devised, for example the “PentaChorus”preset of the Roland RSP-550 signal processing module.) In contrast tohomogeneous layered signal processing, FIG. 60 illustrates separableexamples of inhomogeneous layered signal processing. The examples ofFIG. 60 may be used as shown, with selected omissions, or as an archtypefor similar constructions as provided for in the invention. In oneexample a group pickup signal 6001 is applied to a distortion element6011 and a compression element 6013. Each of the output's signals 6104,6005 as well as the original signal are provided to separatespatializers 6012-6015 (for example, chorus, flange, reverb, etc.) whichare then mixed down into a stereo signal 6021 a, 6021 b by an outputmixer 6020. The invention provides for the substitution of any of theelements 6011, 6013, 6012-6015 with other types of audio signalprocessing elements as well as inclusion of additional layers. Asexpansion of the example, individual vibrating element signals fromvibrating elements sharing the aforementioned group pickup can also beused to create additional layers. For example, an individual bass stringsignal 6002 (for example, the 5th and or 6th string of a guitar) may, inparallel, be processed by a pitch-shifter, OC-2/filter, etc. 6016 tocreate a bass note pitch signal 6006 which, in turn, may be presented toa separate spatializer 6017. Further, another individual string signal6003 can be processed in a similar fashion but replacing thepitch-shifter with an emphasis effect for emphasizing a particularmelody or note in a chord.

Because of the larger number of sonic sources that can be staticallydistributed in the sound field, the invention provides for the use oflocation modulation with a wider range of permissible modulation ratesand modulation depths as extremal location modulation behavior is onlypart of the overall spatial sonic structure.

The invention also provides for the use of layered audio signalprocessing in the creation of spatially-distributed timbralrealizations. One example of this would be providing a dedicated stereochorus to each of the six individual string signals of a guitar as wellas a seventh stereo chorus to the group pickup signal, setting eachchorus sweep rate slightly differently and summing the seven stereooutputs into a single stereo mix; this is in fact an example adaptationof the principals illustrated in FIG. 60. Another example is that of thecross-product octave chain to be described later.

The invention provides for the use of waveshaping techniques,particularly those which can be varied in real-time by control signalsand/or hysteretic waveshaping techniques, as signal processing elements.The invention also provides in general for the separate and/orcoordinated control of parameters involved at each audio signalprocessing layer by means of general control signals.

7.2.5 Envelope-Controlled Time and Pitch Modulation

The invention provides for the modulation of the delay time of avariable delay line by a control signal corresponding to the amplitudeenvelope of the delayed signal or an associated signal. This causes atape-recorder speed instability effect correlated to the transientcharacter of the reference signal amplitude envelope; more precisely thepitch changes with the time derivative of the amplitude envelope. Theinvention also provides for the substitution of a variable pitch shiftercontrolled by the time derivative of the same control signal; thisarrangement produces roughly the same effect. In either implementationthe control signal may be first warped by an emphasis non-linearity,control signal delay, and/or other processing functions. The result canbe used in soloing as a climactic effect or in moderation for atransient enhancement. The invention also provides for envelope controlof pitch-shifting without time-differentiating the control signal.

7.2.6 Resonant Distorting Delays

The invention provides for the sitar-like sympathetic/buzz emulationutilizing short high-resonant delays as described in association withFIG. 39 to be used as a more general signal processing element. This isparticularly useful if parameters of the configuration, such as degreesof resonance, degrees of clipping, and modulation depths, can be variedin real-time by control signals.

7.2.7 Hysteretic Waveshaping and Distortion

Hysteresis occurs to some extent in overdriven tube amplifier outputtransformers due to the natural hysteretic properties of the materialsused to make the transformer core. Hysteresis effects in waveformdistortion can create valuable amplitude-varying effects. The inventionprovides for generalized models of hysteresis to be used as awaveshaping technique, and as such a signal processing technique, withparameters of the hysteresis action variable in real-time via controlsignals.

Traditional hysteresis curves for transformers, gears, pseudo-elasticdeformation, etc. are well known (see for example [Visintin]. FIG. 61illustrates an example of a generalized hysteresis model construction asprovided for by the invention. The input/output graph shows examplesymmetric curves that are linear 6102, superlinear 6103, and sub-linear6104 along with the time/amplitude oscillograph of an example appliedwaveform 6110. Other types of symmetric or non-symmetric non-linearitiesmay also be used. A time-derivative operation on the applied signalwaveform 6110 followed by sign detection reveals whether the appliedsignal waveform is at any instant increasing or decreasing. As anexample, the applied signal waveform 6110 would be applied to onenon-linear warping function such as 6103, 6104 when increasing and theother when decreasing, resulting in the waveform made of segments 6113,6114 rather than the waveform 6112 that would have been created by thelinear curve 6102. In order to allow the applied input signal to vary inamplitude and still maintain continuity of the waveform, the inventionprovides for the warping non-linearities to be themselves adaptivelyscaled or otherwise altered based on amplitude information from thecurrent and previous direction reversals, moving average of waveformarea or waveform power, etc. The invention provides for aspects of thehysteresis process, such as curve shapes and degrees of dependency onwaveform history, to be varied in real-time by control parameters.

Hysteretic waveshaping can be of use in layered audio signal processingand spatially-distributed timbral realizations which have been describedabove.

7.3 Audio Signal Synthesis

Referring to FIGS. 1-2, the invention provides for the inclusion ofaudio signal processing elements 125 with the signal routing,processing, and synthesis entity 120. These may include conventionalMIDI synthesizers, analog synthesis elements, or other technologiesparticular to the invention such as cross-product octave chains,hysteretic waveshaping, vowel sound synthesis, etc. The inventionprovides for as many parameters as possible to be variable in real-timeby means of standardized control signal formats. The invention alsoprovides for the synthesis of vibrating element feedback soundscontrolled by control signals and pitch sampling as may be adapted fromthe Boss DF-2 “Distortion/Feedback” product.

7.3.1 Spatially Distributed Timbre Construction

It is possible to create spatially distributed timbre realizations aspart of the audio synthesis process as well as by subsequent signalprocessing (cross-channel modulated delay, multi-layer chorused stereodistortion, phased multi-signal constellation location modulation, etc.)as described earlier. The invention provides for spatially distributedtimbre realizations within synthesis by a variety of methods. Onemethod, found in many commercial synthesizer modules (such as the KorgM3-R, Korg X5DR, and Kawia K4-r, for example), is for the synthesizervoices themselves to involve multiple parallel oscillators and/orsample-players delivered in the stereo or other multi output form. Thissub-section discusses two other methods provided for by the invention.

7.3.1.1 Cross-Product Octave Chain

The many times aforementioned cross-product octave chain involves two ormore octave divider chains whose corresponding outputs are multipliedtogether, with all resulting outputs summed together by a multipleoutput mix-down mixer. The cross-product technique results in a numberof parallel signal outputs with widely differing spectral contents andspectral animation features, and the animation features slow to a haltwhen all fundamental and overtone frequencies of the two input signalsare brought into fixed integer and small integer-ratio multiplicativerelationships. The aforementioned characteristics of the multipleoutputs lend themselves to spatially-distributed timbral realizationssince mixing of the outputs can partition the frequency content andanimation features differently between the final mix-down outputs. Theinvention provides for the incorporation of cross-product octave chainsin audio single synthesis.

FIG. 62 shows an example implementation of a cross-product octave chainparticularly suited to low cost implementation with logic chips orsimple DSP program loops. Two input signals 6201 a, 6201 b are appliedto optional comparators 6202 which convert the applied signals intotwo-value waveforms. These are applied to a chain of octave droppingelements 6203 which here can be implemented in isolation as a chain oftoggle flip-flops and in aggregate as a binary counter. The depth of thechain can include many levels with three to six a useful number oflevels. Each of the resulting output signals at corresponding levels aremultiplied together by multipliers 6204. Signal analysis of the truthtable for various logical operations show that for applied square wavesan EXCLUSIVE-OR operation acts exactly as a unity-gain multiplication(while an AND function acts as a unity-gain multiplication added tohalf-amplitude versions of the two applied square waves), so themultipliers 6204 may be realized by EXCLUSIVE-OR operations. Thisamounts to less than $1 U.S. worth of chips in hardware and a smallamount of code in software. All outputs are provided to an output mixer6205 which produces at least stereo outputs 6206 a, 6206 b as well aspotentially other mix-down outputs 6207.1-6207.n and which in apreferred embodiment may be adjusted in real-time by control signals.

The invention provides for alternate implementations, for exampleomitting the comparators 6202, implementing the octave drop functions6203 with pitch shifters or OC-2/filter technology, and/or implementingthe multipliers 6204 with VCAs or 4-quadrant multiplier operations. Theinvention also provides for expansions to include more than two octavechains.

In the context of audio signal synthesis, the applied signals 6201 a,6201 b may be generated by two oscillators within a single synthesizervoice; these oscillators may be relatively tuned in unison, octaves, ornear-consonant intervals for basic operation, and one of the oscillatorsmay be continuously swept through a range of pitches to create hugeaudio displays of pleasing spectral complexity.

7.3.1.2 Multi-Channel Waveshaping

The invention also provides for spatially distributed timbrerealizations through use of parallel or complementary modulations of aplurality of waveshaping operations by control signals. The outputs ofthe plurality of waveshapers are then mixed into a stereo ormulti-channel output mix.

7.4 Control Signal Routing

The invention provides for extensive control capabilities and as suchrequires sophisticated control routing, processing, and stored programorganization. The capabilities for this provided by the invention aredescribed in the following sub-sections. To illustrate essentialcapabilities the discussion below is stated in terms of commonlyappreciated MIDI messages and conventions, but the invention providesfor these same capabilities to apply to other signal formats in digital,analog, contact closure, entirely software, etc. or any combination.

7.4.1 General Control Signal Switching and Merging

Referring to FIGS. 1-2, input signals directed to control routing andmerging include the control outputs from the instrument entities 100(including foot controllers), control signal extraction elements 128 a,control signal processing elements 123, and control signal synthesiselements 129 b. Still referring to FIGS. 1-2, output signals directedfrom control routing and merging include control inputs to instrumententities 100, all elements within the signal routing, processing, andsynthesis entity 120 (probably excluding the power supply 121), as wellas any external lighting and/or special effects control systems.

Using MIDI messages and conventions as a model, control signals may becarried through cables and subsystems in combinations of multiplexedformats (the sixteen MIDI channels plus the variety of message types)and space-division formats (multiple MIDI cables). In the MIDI contextthe invention provides for control signal routing at the MIDI port(i.e., MIDI cable) level, the MIDI channel level, and the message index(MIDI note numbers, MIDI Continuous Controller numbers, etc.) level.This same hierarchy of routing capabilities would also apply to non-MIDIcontrol signal equivalents. The invention also provides for theprocessing of control signals at any of these levels.

The MidiTemp MIDI processor products are by far the most comprehensivecommercial products known at this writing; they provide full-capabilityport level and channel level routing but only very limited capabilitiesat the message index level. Further, the invention provides for controlswitching and merging functions to preferably be an integrated componentwithin a larger-scale hardware and software construct rather than anoff-the-shelf module.

To aid in using control signals throughout the system, the inventionalso provides for visual indicators of control message value, such LEDbar-graphs which may be accessed through control signal routing.

7.4.2 Multi-Channel Control Signal and Stored Program Handling andOrganization

The invention provides for a flexible control and configurationhierarchy for signal routing, processing, and synthesis entities.

FIG. 63 illustrates an example flexible control and configurationhierarchy for control signal and stored program handling andorganization. In the Figure, all “program” entities (with the potentialexception of the configuration program 6310) are stored programs whichcan be recalled up and swapped under the command of control signals(such as MIDI Program Change commands). Each stored program can in turncase the recall and/or swapping of other stored programs in accordancewith those arrowed lines between pairs of programs as shown in FIG. 63.

Referring to FIG. 63, input control signals 6300.1-6300.n frominstrument entities, foot controllers, etc. are applied to a controlsignal routing and control signal processing environment 6301. Thecontrol signal routing and control signal processing environment 6301may internally be decomposed into separate control signal routing andcontrol signal processing elements or instead be integrated together ina common realization (as is common on many MIDI signal routing andhandling products). The control signal routing and control signalprocessing environment 6301 then distributes output control signals6302.1-6302.m throughout the rest of the system, specifically allclasses of elements depicted in FIG. 63, potentially including controlaspects of itself. Among the elements receiving control signals is aconfiguration program 6310 which potentially provides simply abackground environment defining specific ports, safeguards, and anycommon control distribution frameworks (point to point, broadcast,daisy-chain) should this be necessary. A variety of configurationprograms 6310 may be made available for varying operational modes (forexample stand-alone operation, ganged operation with one or more othersignal routing, processing and synthesis entities, backup modes,diagnostic modes, etc. A more significant receptor of the output controlsignals 6302.1-6302.m is the master program 6320. This stored programcan change the configuration of the control signal routing and controlsignal processing environment 6301 as well as the choice of any of thesubsystem stored control programs 6331-6337 affiliated with sequencing,audio, lighting, etc. These control programs in turn may change theconfiguration of the control signal routing and control signalprocessing environment 6301 (specifically 6331, 6332, 6334, 6335 areshown with this capability in FIG. 63) as well as the choice of any ofthe stored programs in the clusters of specific subsystem elementsrepresented by 6342-6347 as well as the potentially external lightingsubsystem element cluster 6348; specifically these stored programcommand paths are as indicated by the arrows in FIG. 63. Further, thesequencer control program calls up and potentially immediately initiatesreal-time control signal sequences 6341 some of which may also changethe configuration of the control signal routing and control signalprocessing environment 6301 as indicated by the arrowed line in FIG. 63.

It is understood that FIG. 63 serves as an illustrative example and thatthe invention provides for other organizational structures of thisflavor and spirit.

7.5 Control Signal Processing

The invention provides for control signal processing to be included soas to add extensive valuable control capabilities. For convenience thesecontrol signal processing operations are described in terms of MIDI; theinvention provides for these capabilities in other control signalformats as well.

Monodic Operations:

-   -   intelligent harmony (note by note remapping, individually or in        ranges of arbitrary size)    -   note-number to MIDI Continuous Controller values    -   note-velocity to MIDI Continuous Controller values    -   MIDI Continuous Controller values to note number messages    -   MIDI Continuous Controller value transformed by fixed scaling        and offset values    -   MIDI Continuous Controller values (0-127) remapped to arbitrary        mappings by point, by line segment, or by fitted curve segment    -   MIDI Continuous Controller complementary value transformation        (i.e., if received value is “x”, transmitted value is “127-x”)

Message Delay

-   -   message value threshold tests resulting in the issuances of new        messages message value threshold tests resulting in selected        routing choices for the received message.

Polyadic Operations:

-   -   multiplication of MIDI Continuous Controller values    -   scaling and offset of MIDI Continuous Controller values        controlled by other MIDI

Continuous Controller Values

-   -   MIDI Continuous Controller to Note number and Note velocity    -   sequence detection in a received series of MIDI messages,        potentially within a defined time window, resulting in a new        issued message.        7.6 Control Signal Extraction

The invention provides for the extraction and derivation of controlsignals from audio and video signals as described below

7.6.1 Audio Signal to MIDI Note Event

The invention provides for the conversion of received audio signals intonote events as is standardly done in products such as the Roland GP-10,GM-70, and CP-40. The invention also provides for more advancedextractions and derivations as explained below.

7.6.1.1 Envelope Tracking to MIDI

The above conversions of received audio signals into note events as isstandardly done in products such as the Roland GP-10, GM-70, and CP-40have been limited to channel allocation, note number and note velocity.The invention provides for the real-time extraction of amplitudeenvelope information and its conversion to control signals. For example,the amplitude envelope may be used to control a signal processor orsignal pan location. Because the amplitude envelope falls off in atypically exponential way over time while most control structures expectlinear variation, the invention provides for one or more possiblewarpings of the envelope signal, such as logarithm or piece-wise linearconstructs. Further, the invention also provides for high-pass,band-pass emphasis/notching, and low-pass filtering prior to parameterextraction so as to limit unwanted influence of audio signal transientsat the initial execution of a vibrating element or audio synthesizednote.

7.6.1.2 Control Signal Extract from Vibrating Element Overtones

The use of pitch-detecting interfaces for converting the pitchedvibrations of individual vibrating elements into control signals for usewith synthesizers or other musically-oriented signal processing has beenin use for many years, particularly since shortly after the invention ofthe MIDI standard for electronic instrument control. However, suchpitch-detecting interfaces have derived only the fundamental frequencyand overall amplitude of the pitched vibrations of individual vibratingelements of an instrument. The use of filter banks for determining theenergy in course frequency bands for the purposes of controllingmusically-oriented signal processing (i.e., the so-called “vocoder”) isalso known. However, the practice of determining the scale-accuratepitches and amplitudes of individual overtones for the purposes ofcontrolling synthesizers or other musically-oriented signal processingis currently not known.

Current synthesizer interfaces (such the Boss GP-10 for guitars and theZeta products for violins) typically only respond to the fundamentalvibrating pitch and the overall amplitude. Further, amplitude responsesin these current synthesizer interfaces typically only respond to theamplitude at the initial attack of a note and the event where theamplitude of the sustained vibration falls below a certain threshold.

The invention provides for an expansion of traditional synthesizercontrol interfaces for vibrating elements so as to respond to thepitches and amplitudes of higher-order overtone vibrations and issuecontrol signals based on these. By expanding the response of traditionalsynthesizer interfaces for vibrating elements to include continuous timeresponse to fundamental and overtone amplitudes as well as pitches, farmore expressive control over synthesized sound via tracking of vibratingelements can be obtained. For example, plucking or bowing a string invarying locations can be used to control signal processing parameters.

Traditionally, synthesizer interfaces for vibrating elements capturepitch (based on fundamental frequency of vibration) and amplitude,initially when a vibrating element is excited and in some cases as pitchand/or even amplitude changes dynamically. However, this can be expandedto include responses to various higher-order (non-fundamental) harmonicsor other modes of vibration. It is important to note that such a featurecan add tremendous control over conventional synthesizer soundproduction in general situations where vibrating elements are used tocontrol the synthesis of the sound; this is true somewhat in guitars,but much more so in wind and bowed instruments. In using a vibratingelement feedback excitation arrangement for guitars, for example, thistype of control signal extraction may be especially expressive as thefeedback process can create widely-varying harmonic content when handstouch vibrating elements in feedback excitation or by varying theexcitation feedback characteristics (via signal processing within thefeedback loop). Because of the dynamic overtone characteristics ofexciting vibrating elements in feedback loops, it is of interest toexpand traditional synthesizer interfaces for vibrating elements torespond to the pitches and amplitudes of these higher-order overtonevibrations. The significant synergistic value of the combination ofvibrating element excitation and overtone tracking control signalextraction are also recognized as part of the invention.

There are various ways to accomplish such overtone tracking. In general,it is much easier for instruments whose elements vibrate at fixedpitches with a known overtone series. In these instruments, the overtonefrequencies of a given vibrating element are also known in advance. FIG.64 shows an example method for the generation of control signals fromfundamental and overtone information in a signal from a vibratingelement of fixed known pitch. If the pitch is both fixed and known, thesignal from each vibrating element can be filtered by a suite ofband-pass filters 6402.1-6402.h, each separately tuned to the knownfrequency of an individual mode of vibration for that particularelement. The output of each filter can be fed to a dedicated amplitudefollower 6403.1-6403.h. Each amplitude follower output can be used tocreate a separate parameter which can be assigned to an outgoing controlsignal 6405 via parameter mapping operation 6404.

The invention provides for the combining and/or processing offundamental and overtone information in creating yet other derivedcontrol signals. FIG. 65 shows combining and/or processing offundamental and overtone information obtained from a vibrating elementsignal prior to parameter assignment to control signals. As shown inFIG. 65, the outputs of groups of amplitude detectors associated with agiven vibrating element can be combined and/or processed 6406 beforemapping to final parameters prior to control signal assignment. Forexample, different weighted sums can be used to control the amplitude ofa synthesized signal (say a uniform averaging, or sum-of-squaresaveraging) than would be used to control the cut-off frequency of asubtractive filter (here, weighting the higher modes of vibration morestrongly would make the synthesis mimic the vibrating element's harmonicbalance; weighting the lower more strongly would make the synthesiscomplement the vibrating element's harmonic balance, etc.).

In the case where the vibrating elements do not vibrate at a fixed pitchbut still obey a known overtone relationship, a slightly more involvedversion of the same mechanism can also be used. Note that such animplementation is hardly limited to feedback systems and could be usedin general guitar and violin synthesizer interfaces for new depths ofperformance control. In addition, because variation in overtone seriesdynamics is an essential factor in singing and in percussioninstruments, such a technology opens important new doors for synthesizerovertone-nuance tracking for voice and percussion instruments. Insinging in particular, the relative-amplitudes of the first threeharmonics (largely the first two, actually) determine the choice of sungvowel; as a result, this technology allows synthesizers to track theformants of vowel production in the human voice.

FIG. 66 shows an example implementation of an adaptive method fortracking overtones for a variable-pitch vibrating element with knownovertone series. The method is largely the same as the fixed-pitch case,but with some added steps. The additional steps employed are to firstuse a traditional pitch detector 6407 (as used with conventional MIDIguitar/violin/voice interfaces) to determine the fundamental pitch, thenuse this pitch information plus an overtone series model of thevibrating element 6400 to position the frequencies of the individualband-pass filters 6402.1-6402.h and amplitude followers 6403.1-6403.h.

In a preferred implementation of this approach, the detected pitchinformation provided by the pitch detector 6407 is fed to a model-basedovertone series calculator 608. The model-based overtone seriescalculator 155 generates the control signals required to individuallycenter each of the plurality of band-pass filters 6402.1-6402.h. Themodel-based overtone series calculator 6408 is also used to generateovertone frequency information for use in any combining or processing ofthe extracted overtone amplitude information and in the parametermapping 6404 to final output control signals.

7.6.2 Pluck Direction to MIDI

The invention provides for the extraction of plucking direction (as onan instrument string) of arbitrary vibrating element and creating acontrol signal from it. Core technologies for detecting pluck directiontypically include separate analysis of the signals from a 2-coil humbuckpickup and have been implemented in products by Biax and Passaic.Passaic also implemented a method for deriving a control parameter fromwhere a string was plucked between the bridge and the neck. Theinvention provides for these extraction functions to be included in theavailable control extraction capabilities.

7.6.3 Video Motion and Feature Extraction

The invention provides for the extraction of parameters from providedvideo signals as described earlier and creating control signals fromthem. Methods for implementing this have been described earlier,including simple timing tests and video frame grabs analyzed bydedicated systems or personal computer software. The invention alsoprovides for implementations using emerging motion tracking and imagedecomposition methodologies under development for widespread adoption indigital video compression standards such as MPEG-4 (see for example[Hara; Borinans].

7.6.4 Control Signal Pattern Recognition

The invention provides for the recognition of control signal patterns.Since the result is yet another control signal, this has been treatedearlier in the context of control signal processing.

7.7 Dynamic Control Signal Synthesis

The invention provides for the synthesis of dynamic control signals suchas low-frequency sweep oscillators, particularly those whose parametersmay be controlled in real-time by other control signals. Since anenvelope generator trigger is also a control signal, the generation ofcontrol signal envelopes and slews are also included in this categoryand are provided for by the invention.

7.7.1 MIDI-Controlled Low-Frequency Control Oscillators Ensembles

Low-frequency sweep oscillators, or LFOs, have roles throughout theinvention and have been discussed earlier. In some types of functionsimplemented by specific elements, such as chorus and flangers, the LFOmay be hard associated with the element. The invention provides for thisas well as the remote positioning of the LFO function outside theelement in the case where several elements may be coordinated with thesame LFO. In other types of functions, such as location modulation, itmay be best to control existing elements such as mixers with controlsignals from external LFOs.

The invention provides for a plurality of control signal LFOs to beavailable. The LFOs may be part of a comprehensive system or a separatemodule which can be manufactured and sold for other uses; such a productwould be naturally served by at least MIDI output and input, but mayalso include at least one analog input and/or output. The LFOs providedfor by the invention include multiple phase output capabilities as wellas selections of a variety of waveforms, frequency settings, amplitudesettings and offset settings, all of which may be varied in real-time byyet other control signals. Further, the invention provides for theseparameters to be available under selectable stored program control whichmay be chosen by control signals. Finally, the invention provides forglobal effects across groups of LFOs, such as timing slew of parameterchanges, global scaling, global offsets, etc. These may also includemore complex organizations such as may be require for two-dimensionaland three-dimensional location modulation and the custom construction orsampling of LFO waveforms.

FIG. 67 illustrates an example approach wherein a plurality of LFOs withfeatures as prescribed by the invention may be implemented. Program datafor M different stored programs may be stored in M data structures6700.1-6700.M; these may include specific LFO parameters6701.1.1-6701.M.N for N LFOs. The data structures may also includeglobal information 6702.1-6702.M pertaining to groups of LFOs. Areceived control signal and/or panel control may be used to designatethe selected program 6703 which is recalled and implemented 6704 as aset of default values subject to real-time change by other controlsignals. The LFO data vector structure for each LFO includes a source ofselect information 6711 for choosing whether a given output operates asan independent LFO or as a slave to another LFO specified here (and inso doing becomes a multiple phase output for the chosen sourceoscillator). Another part 6712 of the LFO data vector provides afrequency setting if the LFO is independent and a phase setting ifslaved; should integer-ratio phase-locking be implemented, this part ofthe data structure may be reorganized to include relative frequency andphase settings with respect to the selected master LFO. Another part6713 of the LFO data vector provides selected waveform information,including reference to any user sampled LFO waveforms. Another two parts6714, 6715 of the LFO data vector provide respectively amplitude scalingand offset settings. Additional information, such as the outgoing MIDIchannel and MIDI Continuous Controller number to be used and whatincoming MIDI Continuous Controller messages on what MIDI channel areused to control the aforementioned LFO settings. The global part of thedata 6731, 6732 provides global information for specific settings ofglobal amplitude, offset, parameter, time slew, etc., pertaining tospecific groups of LFOs. The selected information is presented to theLFO engine for execution. The LFO engine in the example implements anindependent LFO by dividing 6751 a system clock signal 6741 by a numberdetermined by the frequency setting 6712. The divided clock signal runsa counter 6752 (here a 128 step counter is illustrated, although higherresolution may be desired). Should the LFO be instead designated, viathe information 6711, to be slave to another LFO, the counter of thatmaster LFO is accessed 6753 and the phase offset information 6712 isused to create a phase offset value which is provided together with theaccessed counter value to an modular adder 6755 which produces theresulting phase-shifted version of the master LFO count value. Theresulting counting sequence produced by this section 6750 is used as theaddress for a waveform lookup table 6756 and/or algorithm; the resultinggenerated periodic signal is then scaled 6757 according to 6714 andoffset in amplitude 6758 by information 6715. The resulting waveformscan be post-processed to provide global amplitude and offset features,or, alternative, mathematical transformations 6731 may be provided onthe information 6714, 6715 before executing it in the LFO engine inelements 6757, 6758.

7.7.2 Controlled Slews, Ramp, and Envelope Generator Elements

The invention provides for slew limiters, ramp generators, and envelopegenerators whose trigger and various parameters may be varied inreal-time by control signals. Slew limiters limit the rate of change ofa control signal to a maximal range which may be set as a parameter andadvantageously varied by control signals. Ramp generators are simplifiedenvelope generators triggered by control signals which ramp between twoor more discrete values or the entire control signal range and do soaccording selected types of dynamics (linear over time, exponential overtime, etc.); the parameters here may be set and advantageously varied bycontrol signals. Envelope generators offer more complex transientwaveforms, typically including at least attack, decay, sustain, andrelease; more complex envelope features including more breakpoints,delays, and segment curve shapes may also be provided. The parametershere may be set and advantageously varied by control signals.

7.8 Lighting Effects and Video Display

The invention provides for extensive control of lighting via controlsignals. Some aspects of lighting as provided for by the invention aredescribed in the sub-sections below.

7.8.1 Light Types

The invention provides for a wide range of types of lighting to becontrolled via control signals. Some example types of lighting providedfor by the invention are described in the sub-sections below.

7.8.1.1 Traditional Fixed

The invention provides for traditional fixed lighting arrangements asshown in FIG. 68. These may include any one or more of overhead lights6802, far-throw lights 6807, foot lights 6803, floor lights aimed upwardor at angles 6804, backlights 6806 behind equipment and risers 6805 andbackdrops 6801.

7.8.1.2 Movable

The invention provides for movable lighting controlled in real-time viacontrol signals. Such lighting can be implemented by attaching lights tomotorized pan/tilt heads as used for video cameras.

7.8.1.3 Instrument Lighting

The invention provides for lighting on instrument entities which may beoperated via control signals. FIG. 69 shows examples of lighting for aguitar 6900 from the bridge 6901, neck 6902, above 6904 and below 6903the picking area, all aimed at illumination effects for the hands. Alsoshown are lights aimed at the audience in the pickup areas 6905, 6906and fret areas 6907, any of which may be aggregate as in 6905, 6906 orsplit out separately for each string as in 6907.

7.8.1.4 Light Sculptures

The invention provides for light sculptures under control of controlsignals. FIG. 70 shows rotating speaker emulation light sculptures. Inone implementation a rotating reflective beacon 7011 reflects gatheredlight from a bulb 7012 and projects it on to a translucent concealingcover 7013 attached to, in this case, a pyramid frame. An alternatearrangement where the translucent outer structure 7023 itself rotates isalso shown about a fixed light bulb 7022 whose cable 7026 fits throughthe bearings 7024, 7025 of a rotating turntable 7021, 7024 driven bygeared 7027, 7028 motor arrangement. The mechanism just described isexploded for explanation and in fact may be readily collapsed bystandard means so as to permit two rotating pyramids 7031, 7032 to bestacked in transparent cubes 7033, 7034. The motor speed and directioncan be controlled by control signals and arranged to operate insynchronization with rotating speaker simulations in audio signalprocessing elements.

FIG. 71 shows light pyramids 7100 made of similar elements 7101-7106,typically without reflectors or directional bulbs, which may be arrangedin arrays 7108. Also shown are light column arrays suggestive of organpipes 7118 or instrument strings 7119. These light columns may be builtwith standard lamps and reflectors 7112, 7114, 7115, colored gels 7111and light transmitting and scattering rods 7110.

Also provided for by the invention are controlled ionize gas turbulencesculptures; these may be used with or without associated video cameras.

7.8.2 General Lighting Control

The invention provides for lights to be used in scene change modes ormodulated by control signals according to:

-   -   animation sequences and subsequence events    -   instrument activity    -   timbre qualities.

Special Instrument lighting effects include:

-   -   audience shock events    -   animation sequences    -   string activity, note following, orchestration following.        7.8.3 Video Signal Routing

Referring to FIGS. 1-2, input signals directed to video routing includethe video outputs from the instrument entities 100, video signalprocessing elements 127, video signal synthesis elements 129 a, andexternal video feeds from, for example, miscellaneous stage cameras,VCRs, etc. Still referring to FIGS. 1-2, output signals directed fromvideo routing include video inputs to instrument entities 100, videosignal processing 127, control signal extraction 128 a, and overallvideo outputs to one or more of any displays, projectors, and/orrecording facilities.

7.8.4 Video Signal Processing

Video signal processing as provided for by the invention would includeoverlays, wipes, fades, blends, solarizations, geometric warping, etc,as much as possible under the control of control signals. Interestingeffects provided for by the invention include the switching, wiping,blending, fading, warping, etc. of various video signals for display inperformance and/or recording under the control of instrument note andamplitude envelope signals.

7.8.5 Video Display

FIG. 72 shows how the invention provides for video projection to be usedto shine down 7201 on the stage 7200, shine horizontally 7202 on toperformers or back-drops. The invention also provides for videoprojectors to be shined on the audience. The invention also provides formovable cameras controlled in real-time via control signals usingmotorized pan/tilt heads as well as motorized zoom/focus lenses.

7.8.6 Video Signal Synthesis

The invention provides for video signal synthesis would includereal-time generation of text message screens, text overlays, vector andraster graphic drawings, vector and raster graphic overlays, andanimations affiliated with numerical dynamics simulation. The inventionalso provides for pre-stored video frames, playback of video clips, andplayback of prestored vector and raster graphics animations. Theinvention provides for these to be controlled by standardized controlsignals, such as MIDI, and as such would typically involve both storedprogram control and parameterized control. These functions may berealized with a conventional personal computer fitted with video cardand MIDI interface as well as by dedicated hardware.

8 Example Envisioned Applications

A few example envisioned applications of the invention are now provided.

8.1 Add-on Modules for Existing Instruments

This gives rise to a whole new marketplace for new instruments,instrument retrofit kits, and music signal processor units which caninteract with external amplifiers, signal processing, and MIDIsynthesizer units.

8.2 Creation of Enhanced Electronic Vibrating Element Instruments

With the first technique described within this patent, the moretraditional acoustically-excited “controlled feedback” effects caneasily be obtained, via electromagnetic excitation, with standard parts.Specialization of the parts can provide additional features. Thetechnique can also be applied to any instrument where sound is produceby vibrating ferromagnetic material, e.g., African mbiras, violins,xylophones, etc.

With the second technique described in this patent, conventional signalprocessing can be used on each string signal to create “generalizedpedal steel guitars,” multi-modal Indian sitars (where drone andsympathetic strings can be electronically retuned while playing),spatially animated string sounds within a stereophonic or spatial soundfield, and mixed timbre instruments where different signal processingmethods are applied to each string. The technique can also be applied toany instrument where vibration of individual sound-producing elementscan be electronically captured by isolated transducers (electromagnetic,optical, Hall-effect, etc.), such as nylon-stringed instruments,marimbas, African mbiras, violins, etc.

By combining these two new techniques with appropriate signalprocessing, a very powerful environment for multi-stringed electronicinstruments can be created. Individual strings can be singled out forfeedback operation while others operate without feedback, and allstrings can be electronically pitch-shifted as needed in a performance.The results allow a performer a greater degree of polyphonic control,using mechanical (neck, frets, fingers, picks, movable tailpieces, pedaltuning changers, etc.) or electronic means for both string excitationand pitch control, with individual string outputs available forsynthesizer interfaces.

Any to all of the above can be built into an individual instrument.Alternatively, an instrument interface can be created and most signalprocessing can be remotely located from the instrument, connecting to itvia this interface. If this interface is standardized across multipleinstruments, then common signal processing equipment environment can beused across a wide variety of instruments (metal-stringed andnylon-stringed guitars, basses, violins, steel guitars, sitars, mbiras,etc.). This gives rise to a whole new marketplace for new instruments,instrument retrofit kits, and music signal processor units which caninteract with external amplifiers, signal processing, and MIDIsynthesizer units.

All publications and patent applications mentioned in this specificationare herein incorporated by reference to the same extent as if eachindividual publication or patent application was specifically andindividually indicated to be incorporated by reference. The inventionnow being fully described, it will be apparent to one of ordinary skillin the art that many changes and modifications can be made theretowithout departing from its spirit or scope.

REFERENCES CITED

The following references are cited in this patent application using theformat of the first one or two authors last name(s) within squarebrackets “[ ]”, multiple references within a pair of square bracketsseparated by semicolons “;”

-   [Bicanic] Bicanic, Dane, ed. Photoacoustic and Photothermal    Phenomena III, Vol. 69, Springer-Verlag, Berlin Heidelberg, 1992;-   [Ronse] Ronse, Christian and Devijver, Pierre A., Connected    Components in Binary Images: the Detection Problem, John Wiley &    Sons Inc. New York, 1984;-   [Viberg] Viberg, Mats, Subspace Fitting Concepts in Sensor Array    Processing, Linkoping Studies in Science and Technology.    Dissertations No. 27 Linkoping, Sweden 1989;-   [Shapiro] Shapiro, Larry S, Affine Analysis of Image Sequences,    Cambridge University Press, 1995;-   [Appelman] Appelman, D. Ralph, The Science of Vocal Pedagogy, Theory    and Applications, Indiana University Press, Bloomington, 1967;-   [Murphy] Murphy, Aamodt, Spicer and Royce (eds.) Photoacoustic and    Photothermal Phenomena II, Vol. 62, Springer-Verlag, Berlin    Heidelberg, 1990;-   [Winckel] Winckel, Fritz, Music Sound, and Sensation: A Modern    Exposition, Dover, New York, 1967;-   [Hara] Hara, Yoshiko “Matsushita demos multilayer MPEG-4    compression”, Electronic Engineering Times, Apr. 19, 1999;-   [Lusher] Lusher, E. (unk) “Photoacoustic effect: principles and    applications: proceedings of the First International Conference on    the Photoacoustics,” Vieweg, 1984;-   [Wang] Wang, X-J, and Nicolis, G. “Bifurcation Phenomena in Coupled    Chemical Oscillators: Normal Form Analysis and Numerical    Simulations”, Physica 26D (1987) 140-155 North-Holland, Amsterdam;-   [Kuramoto] Kuramoto, Yoshiki, “Rhythms and Turbulence in Populations    of Chemical Oscillators”, Physica 106A (1981) 128-143 North-Holland;-   [Epstein] Epstein, Irving R, “New Chemical Oscillators”,    Non-Equilibrium Dynamics in Chemical Systems, Springer-Verlag,    Berlin/Heidelberg 1984;-   [Gray] Gray, Peter, and Scott, Stephen K, Chemical Oscillations and    Instabilities, Clarendon Press, Oxford 1990;-   [Epstein] Epstein, “The Search for New Chemical Oscillators,” in    Chemical Instabilities, Nicolis and Baras (eds.), D. Reidel    Publishing, Dordrecht/Boston/Lancaster, 1984;-   [Nicolis] Nicolis, G and Baras, F (eds.), Chemical Instabilities, D.    Reidel Publishing, Dordrecht/Boston/Lancaster;-   [Tyson] Tyson, John J., The Belousov-Zhabotinskii Reaction,    Springer-Verlag, Berlin/Heidelberg;-   [Nicolis] Nicolis, G., “Nonlinear Phenomena in Chemistry”,    Introduction to Nonlinear Science, Cambridge University Press, Great    Britain 1995;-   [Orban] Orban, M., Kurin-Csorgei, K., Zhabotinsky, A, and Epstein,    I, “New Indicators for Visualizing Pattern Formation in Uncatalyzed    Bromate Oscillatory Systems”, J. Am. Chem. Soc. 1998, 120,    1146-1150;-   [Field] Field, Richard J., and Noyes, Richard M “Oscillations in    Chemical Systems IV Limit Cycle Behavior in a Model of a Real    Chemical Reaction” J. Chem. Physics Vol. 60 No. 5 Mar. 1974;-   [Nagy-Ungvarai] Nagy-Ungvarai, Z. and Hess, Benno “Control of    Dynamic Pattern Formation in the Belousov-Zhabotinsky Reaction”    Physica D (1991) North-Holland;-   [Ruoff] Ruoff, P. “How Bromate Oscillators Are Controlled”, Am.    Chem. Society 1988;-   [Bormans] Bormans, Jan “MPEG-4 systems need specialized CPUs”,    Electronic Engineering Times, Jan. 25, 1999; and-   [Visintin] Visintin, A (ed.), Models of Hysteresis, Longman    Scientific & Technical, Harlow, England, 1993.

1. A system for enriching the timbre of audio signals, said systemcomprising: a processor adapted to process an incoming audio signalcomprising a signal attribute that is measurable over time, wherein saidincoming audio signal is processed by measuring said signal attribute toproduce an internal hysteretic control signal, wherein said measuring ofsaid signal attribute is performed according to one of a selectableplurality of control signal generation algorithms; a waveshaper adaptedto generate an outgoing waveshaped audio signal by mapping said incomingaudio signal to a particular outgoing waveshaped audio signal, whereinsaid mapping is controlled by said internal hysteretic control signal;and an interface providing said outgoing waveshaped audio signal to anexternal audio system.
 2. The system according to claim 1, wherein saidmapping of said incoming audio signal to said particular outgoingwaveshaped audio signal is additionally controlled by an incomingmapping control signal.
 3. The system according to claim 2, wherein saidincoming mapping control signal comprises a signal of MIDI format. 4.The system according to claim 1, wherein said mapping is one of aplurality of mappings, wherein each mapping of said plurality ofmappings provides for the generation of a corresponding plurality ofdifferent outgoing waveshaped audio signals.
 5. The system according toclaim 4, wherein at least one mapping of said plurality of mappingscomprises a nonlinear characteristic.
 6. The system according to claim1, wherein an incoming hysteresis control signal is used to select aparticular control signal generation algorithm from said plurality ofcontrol signal generation algorithms.
 7. The system according to claim6, wherein said incoming hysteresis control signal comprises a signal ofMIDI format.
 8. The system according to claim 1, wherein said systemprovides one signal processing layer of a multi-layered signalprocessing system.
 9. The system according to claim 1, wherein saidsystem is incorporated into a spatially-distributed timbral realizationsystem.
 10. A method for enriching the timbre of audio signals, saidmethod comprising: receiving an incoming audio signal comprising asignal attribute, wherein said signal attribute is measurable over time;processing said incoming audio signal by measuring said signal attributeto produce an internal hysteretic control signal, wherein said measuringof said signal attribute is performed according to one of a selectableplurality of control signal generation algorithms; generating anoutgoing waveshaped audio signal by mapping said incoming audio signalto a particular outgoing waveshaped audio signal, wherein said mappingis controlled by said internal hysteretic control signal; and providingsaid outgoing waveshaped audio signal to an external audio system. 11.The method according to claim 1, wherein said mapping of said incomingaudio signal to said particular outgoing waveshaped audio signal isadditionally controlled by an incoming mapping control signal.
 12. Themethod according to claim 2, wherein said incoming mapping controlsignal comprises a signal of MIDI format.
 13. The method according toclaim 1, wherein said mapping is one of a plurality of mappings, whereineach mapping of said plurality of mappings provides for the generationof a corresponding plurality of different outgoing waveshaped audiosignals.
 14. The method according to claim 4, wherein at least onemapping of said plurality of mappings comprises a nonlinearcharacteristic.
 15. The method according to claim 1, wherein an incominghysteresis control signal is used to select a particular control signalgeneration algorithm from said plurality of control signal generationalgorithms.
 16. The method according to claim 15, wherein said incominghysteresis control signal comprises a signal of MIDI format.
 17. Themethod according to claim 1, wherein said method is implemented withinone signal processing layer of a multi-layered signal processing system.18. The method according to claim 1, wherein said method is implementedwithin a spatially-distributed timbral realization system.